Re: [DISCUSS] KIP-516: Topic Identifiers

[Jun Rao]

Hi, Justine,

Thanks for the summary. Just a few minor comments blow.

30.  {"name": "id", "type": "string", "doc": "version id"}}: The doc should
say UUID. The issue seems unfixed.

40. Since UUID is public facing, could you include its definition?

41. StopReplicaResponse still includes the topic field.

42. "It is unnecessary to include the name of the topic in the following
Request/Response calls" It would be useful to include all modified requests
(e.g. produce) in the list below.

Thanks,

Jun


On Wed, Sep 30, 2020 at 10:17 AM Justine Olshan <jols...@confluent.io>
wrote:

> Hello again,
>
> I've taken some time to discuss some of the remaining points brought up by
> the previous emails offline. Here are some of the conclusions.
>
> 1. Directory Structure:
> There was some talk about whether the directory structure should be changed
> to replace all topic names with topic IDs.
> This will be a useful change, but will prevent downgrades. It will be best
> to wait until a major release, likely alongside KIP-500 changes that will
> prevent downgrades. I've updated the KIP to include this change with the
> note about migration and deprecation.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-log.dirlayout
>
> 2. Partition Metadata file
> There was some disagreement about the inclusion of the partition metadata
> file.
> This file will be needed to persist the topic ID, especially while we still
> have the old directory structure. Even after the changes, this file can be
> useful for debugging and recovery.
> Creating a single mapping file from topic names to topic IDs was
> considered, but ultimately scrapped as it would not be as easy to maintain.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-PartitionMetadatafile
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-PersistingTopicIDs
>
> 3. Produce Protocols
> After some further discussion, this replacing the topic name with topic ID
> in these protocols has been added to the KIP.
> This will cut down on the size of the protocol. Since changes to fetch are
> included, it does make sense to update these protocols.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-Produce
>
> 4. KIP-500 Compatibility
> After some discussion with those involved with KIP-500, it seems best to
> use a sentinel topic ID for the metadata topic that is used before the
> first controller election.
> We had an issue where this metadata topic may not be assigned an ID before
> utilizing the Vote and Fetch protocols. It was decided to reserve a unique
> ID for this topic to be used until the controller could give the topic a
> unique ID.
> Having the topic keep the sentinel ID (not be reassigned to a unique ID)
> was considered, but it seemed like a good idea to leave the option open for
> the metadata topic to have a unique ID in cases where it would need to be
> differentiated from other clusters' metadata topics. (ex. tiered storage)
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-CompatibilitywithKIP-500
>
> I've also split up the KIP into sub-tasks on the JIRA. Hopefully this will
> provide a better idea about what tasks we have, and eventually provide a
> place to see what's done and what is left.
> If there is a task I am missing, please let me know!
> https://issues.apache.org/jira/browse/KAFKA-8872
>
> Of course, these decisions are not set in stone, and I would love to hear
> any feedback.
>
> Thanks,
> Justine
>
> On Mon, Sep 28, 2020 at 11:38 AM Justine Olshan <jols...@confluent.io>
> wrote:
>
> > Hello all,
> >
> > I just wanted to follow up on this discussion. Did we come to an
> > understanding about the directory structure?
> >
> > I think the biggest question here is what is acceptable to leave out due
> > to scope vs. what is considered to be too much tech debt.
> > This KIP is already pretty large with the number of changes, but it also
> > makes sense to do things correctly, so I'd love to hear everyone's
> thoughts.
> >
> > Thanks,
> > Justine
> >
> > On Fri, Sep 25, 2020 at 8:19 AM Lucas Bradstreet <lu...@confluent.io>
> > wrote:
> >
> >> Hi Ismael,
> >>
> >> If you do not store it in a metadata file or in the directory structure
> >> would you then
> >> require the LeaderAndIsrRequest following startup to give you some
> notion
> >> of
> >> topic name in memory? We would still need this information for the older
> >> protocols, but
> >> perhaps this is what's meant by tech debt.
> >>
> >> Once we're free of the old non-topicID requests then I think you
> wouldn't
> >> need to retain the topic name.
> >> I think the ability to easily look up topic names associated with
> >> partition
> >> directories would still be missed
> >> when diagnosing issues, though maybe it wouldn't be a deal breaker with
> >> the
> >> right tooling.
> >>
> >> Thanks,
> >>
> >> Lucas
> >>
> >> On Fri, Sep 25, 2020 at 7:55 AM Ismael Juma <ism...@juma.me.uk> wrote:
> >>
> >> > Hi Lucas,
> >> >
> >> > Why would you include the name and id? I think you'd want to remove
> the
> >> > name from the directory name right? Jason's suggestion was that if you
> >> > remove the name from the directory, then why would you need the id
> name
> >> > mapping file?
> >> >
> >> > Ismael
> >> >
> >> > On Thu, Sep 24, 2020 at 4:24 PM Lucas Bradstreet <lu...@confluent.io>
> >> > wrote:
> >> >
> >> > > > 2. Part of the usage of the file is to have persistent storage of
> >> the
> >> > > topic
> >> > > ID and use it to compare with the ID supplied in the LeaderAndIsr
> >> > Request.
> >> > > There is some discussion in the KIP about changes to the directory
> >> > > structure, but I believe directory changes were considered to be out
> >> of
> >> > > scope when the KIP was written.
> >> > >
> >> > >
> >> > > Yeah, I was hoping to get a better understanding of why it was taken
> >> out
> >> > of
> >> > > scope. Perhaps Lucas Bradstreet might have more insight about the
> >> > decision.
> >> > > Basically my point is that we have to create additional
> infrastructure
> >> > here
> >> > > to support the name/id mapping, so I wanted to understand if we just
> >> > > consider this a sort of tech debt. It would be useful to cover how
> we
> >> > > handle the case when this file gets corrupted. Seems like we just
> >> have to
> >> > > trust that it matches whatever the controller tells us and rewrite
> it?
> >> > >
> >> > >
> >> > > Hi Jason, Justine,
> >> > >
> >> > > My thought process is that we will likely need the metadata file
> >> whether
> >> > we
> >> > > rename the directories or not.
> >> > > Linux supports filenames of up to 255 bytes and that would not be
> >> enough
> >> > to
> >> > > support a directory name
> >> > >  including both the name and topic ID. Given that fact, it seemed
> >> > > reasonable to add the metadata file
> >> > > and leave the directory rename until some time in the future
> (possibly
> >> > > never).
> >> > >
> >> > > If the file does get corrupted I think you're right. We would either
> >> have
> >> > > to trust it matches what the controller tells us
> >> > >  or error out and let an administrator resolve it by checking across
> >> > > replicas for consistency.
> >> > >
> >> > > Lucas
> >> > >
> >> > >
> >> > > On Thu, Sep 24, 2020 at 3:41 PM Jason Gustafson <ja...@confluent.io
> >
> >> > > wrote:
> >> > >
> >> > > > Thanks Justine. Responses below:
> >> > > >
> >> > > > > 1. Yes, the directory will still be based on the topic names.
> >> > > > LeaderAndIsrRequest is one of the few requests that will still
> >> contain
> >> > > the
> >> > > > topic name. So I think we have this covered. Sorry for confusion.
> >> > > >
> >> > > > Ah, you're right. My eyes passed right over the field.
> >> > > >
> >> > > > > 2. Part of the usage of the file is to have persistent storage
> of
> >> the
> >> > > > topic
> >> > > > ID and use it to compare with the ID supplied in the LeaderAndIsr
> >> > > Request.
> >> > > > There is some discussion in the KIP about changes to the directory
> >> > > > structure, but I believe directory changes were considered to be
> >> out of
> >> > > > scope when the KIP was written.
> >> > > >
> >> > > > Yeah, I was hoping to get a better understanding of why it was
> taken
> >> > out
> >> > > of
> >> > > > scope. Perhaps Lucas Bradstreet might have more insight about the
> >> > > decision.
> >> > > > Basically my point is that we have to create additional
> >> infrastructure
> >> > > here
> >> > > > to support the name/id mapping, so I wanted to understand if we
> just
> >> > > > consider this a sort of tech debt. It would be useful to cover how
> >> we
> >> > > > handle the case when this file gets corrupted. Seems like we just
> >> have
> >> > to
> >> > > > trust that it matches whatever the controller tells us and rewrite
> >> it?
> >> > > >
> >> > > > > 3. I think this is a good point, but I again I wonder about the
> >> scope
> >> > > of
> >> > > > the KIP. Most of the changes mentioned in the KIP are for
> supporting
> >> > > topic
> >> > > > deletion and I believe that is why the produce request was listed
> >> under
> >> > > > future work.
> >> > > >
> >> > > > That's fair. I brought it up since `Fetch` is already included. If
> >> > we've
> >> > > > got `Metadata` and `Fetch`, seems we may as well do `Produce` and
> >> save
> >> > an
> >> > > > extra kip. No strong objection though if you want to leave it out.
> >> > > >
> >> > > >
> >> > > > -Jason
> >> > > >
> >> > > >
> >> > > > On Thu, Sep 24, 2020 at 3:26 PM Justine Olshan <
> >> jols...@confluent.io>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Jason,
> >> > > > >
> >> > > > > Thanks for your comments.
> >> > > > >
> >> > > > > 1. Yes, the directory will still be based on the topic names.
> >> > > > > LeaderAndIsrRequest is one of the few requests that will still
> >> > contain
> >> > > > the
> >> > > > > topic name. So I think we have this covered. Sorry for
> confusion.
> >> > > > >
> >> > > > > 2. Part of the usage of the file is to have persistent storage
> of
> >> the
> >> > > > topic
> >> > > > > ID and use it to compare with the ID supplied in the
> LeaderAndIsr
> >> > > > Request.
> >> > > > > There is some discussion in the KIP about changes to the
> directory
> >> > > > > structure, but I believe directory changes were considered to be
> >> out
> >> > of
> >> > > > > scope when the KIP was written.
> >> > > > >
> >> > > > > 3. I think this is a good point, but I again I wonder about the
> >> scope
> >> > > of
> >> > > > > the KIP. Most of the changes mentioned in the KIP are for
> >> supporting
> >> > > > topic
> >> > > > > deletion and I believe that is why the produce request was
> listed
> >> > under
> >> > > > > future work.
> >> > > > >
> >> > > > > 4. This sounds like it might be a good solution, but I will need
> >> to
> >> > > > discuss
> >> > > > > more with KIP-500 folks to get the details right.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Justine
> >> > > > >
> >> > > > > On Thu, Sep 24, 2020 at 12:30 PM Jason Gustafson <
> >> ja...@confluent.io
> >> > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi Justine,
> >> > > > > >
> >> > > > > > Thanks for picking up this work. I have a few
> >> questions/comments:
> >> > > > > >
> >> > > > > > 1. It sounds like the directory structure is still going to be
> >> > based
> >> > > on
> >> > > > > > topic names. Do I have that right? One complication is that
> the
> >> > > > > > LeaderAndIsr request does not include the topic name any
> longer.
> >> > This
> >> > > > > means
> >> > > > > > that a replica must wait for the UpdateMetadata request to
> >> arrive
> >> > > with
> >> > > > > the
> >> > > > > > topic name mapping before it can create the directory. I
> wonder
> >> if
> >> > it
> >> > > > > would
> >> > > > > > be simpler to avoid assumptions on the ordering of
> >> UpdateMetadata
> >> > and
> >> > > > let
> >> > > > > > LeaderAndIsr include the topic name as well. Feels like we are
> >> not
> >> > > > saving
> >> > > > > > that much by excluding it.
> >> > > > > >
> >> > > > > > 2. On a related note, it seems that the reason we have the
> >> > partition
> >> > > > > > metadata file is because we are not changing the directory
> >> > structure.
> >> > > > We
> >> > > > > > need it so that we remember which directories map to which
> topic
> >> > id.
> >> > > I
> >> > > > > > think it would be useful to add some detail about why changing
> >> the
> >> > > > > > directory layout was rejected. Basically trying to understand
> >> how
> >> > > much
> >> > > > > > effort we are saving by skipping this step if we have to
> >> implement
> >> > > this
> >> > > > > > additional bookkeeping. One concern I have for example is that
> >> the
> >> > > > > > partition metadata file gets corrupt and then all bets are off
> >> as
> >> > far
> >> > > > as
> >> > > > > > topic consistency.
> >> > > > > >
> >> > > > > > 3. Is it worth adding support now for topic ids in the
> `Produce`
> >> > > > request
> >> > > > > > now? Overhead is mentioned as one of the motivations and the
> >> best
> >> > > APIs
> >> > > > to
> >> > > > > > get savings from are `Produce` and `Fetch`.
> >> > > > > >
> >> > > > > > 4. When it comes to bootstrapping the metadata topic with
> >> respect
> >> > to
> >> > > > > > KIP-500, one suggestion would be to reserve a sentinel topic
> ID
> >> > which
> >> > > > can
> >> > > > > > be used initially by new brokers. When the first controller is
> >> > > elected,
> >> > > > > it
> >> > > > > > can assign a topicId to the metadata topic. This is a bit
> >> similar
> >> > to
> >> > > > how
> >> > > > > we
> >> > > > > > were planning to generate the clusterId.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Jason
> >> > > > > >
> >> > > > > >
> >> > > > > > On Thu, Sep 24, 2020 at 11:10 AM Jun Rao <j...@confluent.io>
> >> wrote:
> >> > > > > >
> >> > > > > > > Hi, Justine,
> >> > > > > > >
> >> > > > > > > Thanks for the updated KIP. A few more comments below.
> >> > > > > > >
> >> > > > > > > 30.  {"name": "id", "type": "string", "doc": "version id"}}:
> >> The
> >> > > doc
> >> > > > > > should
> >> > > > > > > say UUID.
> >> > > > > > >
> >> > > > > > > 31. LeaderAndIsrResponse v5 and StopReplicaResponse v4 : It
> >> seems
> >> > > > there
> >> > > > > > is
> >> > > > > > > no need to add topic_id at partitions level.
> >> > > > > > >
> >> > > > > > > 32. Regarding partition metadata file. Perhaps the key can
> be
> >> a
> >> > > > single
> >> > > > > > > word, sth like the following.
> >> > > > > > > version: 0
> >> > > > > > > topic_id: 46bdb63f-9e8d-4a38-bf7b-ee4eb2a794e4
> >> > > > > > >
> >> > > > > > > 33. Another tricky thing that I realized is how to support
> the
> >> > > > metadata
> >> > > > > > > topic introduced in KIP-595. It's a bit tricky to assign a
> >> UUID
> >> > to
> >> > > > the
> >> > > > > > > metadata topic since we have a chicken-and-egg problem. The
> >> > > > controller
> >> > > > > > > needs to persist the UUID in the metadata topic in order to
> >> > assign
> >> > > > one
> >> > > > > > > successfully, but the metadata topic is needed to elect a
> >> > > controller
> >> > > > > > > (KIP-631). So, this probably needs a bit more thought.
> >> > > > > > >
> >> > > > > > > Jun
> >> > > > > > >
> >> > > > > > > On Thu, Sep 24, 2020 at 3:04 AM Ismael Juma <
> >> ism...@juma.me.uk>
> >> > > > wrote:
> >> > > > > > >
> >> > > > > > > > Also, can we provide more details on how the Partition
> >> Metadata
> >> > > > file
> >> > > > > > will
> >> > > > > > > > be used?
> >> > > > > > > >
> >> > > > > > > > Ismael
> >> > > > > > > >
> >> > > > > > > > On Thu, Sep 24, 2020 at 3:01 AM Ismael Juma <
> >> ism...@juma.me.uk
> >> > >
> >> > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi Justine,
> >> > > > > > > > >
> >> > > > > > > > > I think we need to update the "Rejected Alternatives"
> >> section
> >> > > to
> >> > > > > take
> >> > > > > > > > into
> >> > > > > > > > > account that the proposal now removes the topic name
> from
> >> the
> >> > > > fetch
> >> > > > > > > > > request. Also, if we are removing it from the Fetch
> >> request,
> >> > > does
> >> > > > > it
> >> > > > > > > make
> >> > > > > > > > > sense not to remove it from similar requests like
> >> > > > > ListOffsetRequest?
> >> > > > > > > > >
> >> > > > > > > > > Ismael
> >> > > > > > > > >
> >> > > > > > > > > On Thu, Sep 24, 2020 at 2:46 AM David Jacot <
> >> > > dja...@confluent.io
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > >> Hi Justine,
> >> > > > > > > > >>
> >> > > > > > > > >> Thanks for the KIP. I finally had time to read it :).
> It
> >> is
> >> > a
> >> > > > > great
> >> > > > > > > > >> improvement.
> >> > > > > > > > >>
> >> > > > > > > > >> I have a few comments/questions:
> >> > > > > > > > >>
> >> > > > > > > > >> 1. It seems that the schema of the StopReplicaRequest
> is
> >> > > > slightly
> >> > > > > > > > >> outdated.
> >> > > > > > > > >> We
> >> > > > > > > > >> did some changes as part of KIP-570. V3 is already
> >> organized
> >> > > by
> >> > > > > > > topics.
> >> > > > > > > > >>
> >> > > > > > > > >> 2. I just want to make sure that I understand the
> >> > > reconciliation
> >> > > > > > > > >> logic correctly. When
> >> > > > > > > > >> an "INCREMENTAL" LeaderAndIsr Request is received, the
> >> > broker
> >> > > > will
> >> > > > > > > also
> >> > > > > > > > >> reconcile
> >> > > > > > > > >> when the local uuid does not match the uuid in the
> >> request,
> >> > > > right?
> >> > > > > > In
> >> > > > > > > > this
> >> > > > > > > > >> case, the
> >> > > > > > > > >> local log is staged for deletion.
> >> > > > > > > > >>
> >> > > > > > > > >> 3. In the documentation of the `
> >> delete.stale.topic.delay.ms
> >> > `
> >> > > > > > config,
> >> > > > > > > it
> >> > > > > > > > >> says "When a
> >> > > > > > > > >> FULL LeaderAndIsrRequest is received..." but I suppose
> it
> >> > > > applies
> >> > > > > to
> >> > > > > > > > both
> >> > > > > > > > >> types.
> >> > > > > > > > >>
> >> > > > > > > > >> Best,
> >> > > > > > > > >> David
> >> > > > > > > > >>
> >> > > > > > > > >> On Thu, Sep 24, 2020 at 1:40 AM Justine Olshan <
> >> > > > > > jols...@confluent.io>
> >> > > > > > > > >> wrote:
> >> > > > > > > > >>
> >> > > > > > > > >> > Hi Jun,
> >> > > > > > > > >> >
> >> > > > > > > > >> > Thanks for the comments. I apologize for some of the
> >> typos
> >> > > and
> >> > > > > > > > >> confusion.
> >> > > > > > > > >> > I’ve updated the KIP to fix some of the issues you
> >> > > mentioned.
> >> > > > > > > > >> >
> >> > > > > > > > >> > 20.2 I’ve changed the type to String.
> >> > > > > > > > >> > 20.1/3 I’ve updated the TopicZNode to fix formatting
> >> and
> >> > > > reflect
> >> > > > > > the
> >> > > > > > > > >> latest
> >> > > > > > > > >> > version before this change.
> >> > > > > > > > >> >
> >> > > > > > > > >> > 21. You are correct and I’ve removed this line. I’ve
> >> also
> >> > > > added
> >> > > > > a
> >> > > > > > > line
> >> > > > > > > > >> > mentioning an IBP bump is necessary for migration
> >> > > > > > > > >> >
> >> > > > > > > > >> > 22. I think the wording was unclear but your summary
> is
> >> > what
> >> > > > was
> >> > > > > > > > >> intended
> >> > > > > > > > >> > by this line. I’ve updated to make this point more
> >> clear.
> >> > > > > “Remove
> >> > > > > > > > >> deleted
> >> > > > > > > > >> > topics from replicas by sending StopReplicaRequest V3
> >> > before
> >> > > > the
> >> > > > > > IBP
> >> > > > > > > > >> bump
> >> > > > > > > > >> > using the old logic, and using V4 and the new logic
> >> with
> >> > > topic
> >> > > > > IDs
> >> > > > > > > > after
> >> > > > > > > > >> > the IBP bump.”
> >> > > > > > > > >> >
> >> > > > > > > > >> > 23. I’ve removed the unspecified type from the KIP
> and
> >> > > mention
> >> > > > > > that
> >> > > > > > > > IBP
> >> > > > > > > > >> > will be used to handle this request. “IBP will be
> used
> >> to
> >> > > > > > determine
> >> > > > > > > > >> whether
> >> > > > > > > > >> > this new form of the request will be used. For older
> >> > > requests,
> >> > > > > we
> >> > > > > > > will
> >> > > > > > > > >> > ignore this field and default to previous behavior.”
> >> > > > > > > > >> >
> >> > > > > > > > >> > 24. I’ve fixed this typo.
> >> > > > > > > > >> >
> >> > > > > > > > >> > 25. I've added a topics at a higher level for
> >> > > > > LeaderAndIsrResponse
> >> > > > > > > v5,
> >> > > > > > > > >> > StopReplicaResponse v4. I've also changed
> >> > StopReplicaRequest
> >> > > > v4
> >> > > > > to
> >> > > > > > > > have
> >> > > > > > > > >> > topics at a higher level.
> >> > > > > > > > >> >
> >> > > > > > > > >> > 26. I’ve updated forgotten_topics_data--added the
> >> topic ID
> >> > > and
> >> > > > > > > removed
> >> > > > > > > > >> the
> >> > > > > > > > >> > topic name
> >> > > > > > > > >> >
> >> > > > > > > > >> > 27. I’ve decided on plain text, and I’ve added an
> >> example.
> >> > > > > > > > >> >
> >> > > > > > > > >> > 28. I’ve added this idea to future work.
> >> > > > > > > > >> >
> >> > > > > > > > >> > Thanks again for taking a look,
> >> > > > > > > > >> >
> >> > > > > > > > >> > Justine
> >> > > > > > > > >> >
> >> > > > > > > > >> > On Wed, Sep 23, 2020 at 10:28 AM Jun Rao <
> >> > j...@confluent.io>
> >> > > > > > wrote:
> >> > > > > > > > >> >
> >> > > > > > > > >> > > Hi, Justine,
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > Thanks for the response. Made another pass. A few
> >> more
> >> > > > > comments
> >> > > > > > > > below.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > 20. znode schema:
> >> > > > > > > > >> > > 20.1 It seems that {"name": "version", "type":
> "int",
> >> > > "id":
> >> > > > > > > "UUID",
> >> > > > > > > > >> > "doc":
> >> > > > > > > > >> > > "version id"} should be {"name": "version", "type":
> >> > > "int"},
> >> > > > > > > {"name":
> >> > > > > > > > >> > "id",
> >> > > > > > > > >> > > "type": "UUID", "doc": "version id"}.
> >> > > > > > > > >> > > 20.2 The znode format is JSON which doesn't have
> UUID
> >> > > type.
> >> > > > So
> >> > > > > > the
> >> > > > > > > > >> type
> >> > > > > > > > >> > > probably should be string?
> >> > > > > > > > >> > > 20.3 Also, the existing format used seems outdated.
> >> It
> >> > > > should
> >> > > > > > have
> >> > > > > > > > the
> >> > > > > > > > >> > > following format.
> >> > > > > > > > >> > >     Json.encodeAsBytes(Map(
> >> > > > > > > > >> > >       "version" -> 2,
> >> > > > > > > > >> > >       "partitions" -> replicaAssignmentJson.asJava,
> >> > > > > > > > >> > >       "adding_replicas" ->
> >> > > > > addingReplicasAssignmentJson.asJava,
> >> > > > > > > > >> > >       "removing_replicas" ->
> >> > > > > > removingReplicasAssignmentJson.asJava
> >> > > > > > > > >> > >     ).asJava)
> >> > > > > > > > >> > >   }
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > 21. Migration: The KIP says "The migration process
> >> can
> >> > > take
> >> > > > > > place
> >> > > > > > > > >> without
> >> > > > > > > > >> > > an inter-broker protocol bump, as the format stored
> >> in
> >> > > > > > > > >> > > /brokers/topics/[topic] will be compatible with
> older
> >> > > broker
> >> > > > > > > > >> versions."
> >> > > > > > > > >> > > However, since we are bumping up the version of
> >> > > inter-broker
> >> > > > > > > > >> requests, it
> >> > > > > > > > >> > > seems that we need to use IBP for migration.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > 22. The KIP says "Remove deleted topics from
> >> replicas by
> >> > > > > sending
> >> > > > > > > > >> > > StopReplicaRequest V3 for any topics which do not
> >> > contain
> >> > > a
> >> > > > > > topic
> >> > > > > > > > ID,
> >> > > > > > > > >> and
> >> > > > > > > > >> > > V4 for any topics which do contain a topic ID.".
> >> > However,
> >> > > if
> >> > > > > we
> >> > > > > > > use
> >> > > > > > > > >> IBP,
> >> > > > > > > > >> > it
> >> > > > > > > > >> > > seems that the controller will either send
> >> > > > StopReplicaRequest
> >> > > > > V3
> >> > > > > > > > >> > > or StopReplicaRequest V4, but never mixed V3 and V4
> >> for
> >> > > > > > different
> >> > > > > > > > >> topics.
> >> > > > > > > > >> > > Basically, before the IBP bump, V3 will be used.
> >> After
> >> > the
> >> > > > IBP
> >> > > > > > > bump,
> >> > > > > > > > >> > > topicId will be created and V4 will be used.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > 23. Given that we depend on IBP, do we still need
> "0
> >> > > > > > UNSPECIFIED"
> >> > > > > > > > >> > > in LeaderAndIsr?
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > 24. LeaderAndIsrResponse v5 : It still has the
> topic
> >> > > field.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > 25. LeaderAndIsrResponse v5, StopReplicaResponse
> v4:
> >> > Could
> >> > > > we
> >> > > > > > use
> >> > > > > > > > this
> >> > > > > > > > >> > > opportunity to organize the response in 2 levels,
> >> first
> >> > by
> >> > > > > > topic,
> >> > > > > > > > >> then by
> >> > > > > > > > >> > > partition, as most other requests/responses?
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > 26. FetchRequest v13 : Should forgotten_topics_data
> >> use
> >> > > > > topicId
> >> > > > > > > too?
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > 27. "This file can either be plain text (key/value
> >> > pairs)
> >> > > or
> >> > > > > > > JSON."
> >> > > > > > > > >> Have
> >> > > > > > > > >> > we
> >> > > > > > > > >> > > decided which one to use? Also, it would be helpful
> >> to
> >> > > > provide
> >> > > > > > an
> >> > > > > > > > >> > example.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > 28. Future improvement: Another future improvement
> >> > > > opportunity
> >> > > > > > is
> >> > > > > > > to
> >> > > > > > > > >> use
> >> > > > > > > > >> > > topicId in GroupMetadataManager.offsetCommitKey in
> >> the
> >> > > > > > > offset_commit
> >> > > > > > > > >> > topic.
> >> > > > > > > > >> > > This may save some space.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > Thanks,
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > Jun
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > On Wed, Sep 23, 2020 at 8:50 AM Justine Olshan <
> >> > > > > > > > jols...@confluent.io>
> >> > > > > > > > >> > > wrote:
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > > Hi Tom,
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > Thanks for the comment. I think this is a really
> >> good
> >> > > idea
> >> > > > > and
> >> > > > > > > it
> >> > > > > > > > >> has
> >> > > > > > > > >> > > been
> >> > > > > > > > >> > > > added to the KIP under the newly added tooling
> >> > section.
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > Thanks again,
> >> > > > > > > > >> > > > Justine
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > On Wed, Sep 23, 2020 at 3:17 AM Tom Bentley <
> >> > > > > > > tbent...@redhat.com>
> >> > > > > > > > >> > wrote:
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > > Hi Justine,
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > I know you started the vote thread, but on
> >> > re-reading
> >> > > > the
> >> > > > > > KIP
> >> > > > > > > I
> >> > > > > > > > >> > noticed
> >> > > > > > > > >> > > > > that although the topic id is included in the
> >> > > > > > MetadataResponse
> >> > > > > > > > >> it's
> >> > > > > > > > >> > not
> >> > > > > > > > >> > > > > surfaced in the output from `kafka-topics.sh
> >> > > > --describe`.
> >> > > > > > > Maybe
> >> > > > > > > > >> that
> >> > > > > > > > >> > > was
> >> > > > > > > > >> > > > > intentional because ids are intentionally not
> >> really
> >> > > > > > something
> >> > > > > > > > the
> >> > > > > > > > >> > user
> >> > > > > > > > >> > > > > should care deeply about, but it would also
> make
> >> > life
> >> > > > > harder
> >> > > > > > > for
> >> > > > > > > > >> > anyone
> >> > > > > > > > >> > > > > debugging Kafka and this would likely get worse
> >> the
> >> > > more
> >> > > > > > topic
> >> > > > > > > > ids
> >> > > > > > > > >> > got
> >> > > > > > > > >> > > > > rolled out across the protocols, clients etc.
> It
> >> > seems
> >> > > > > > likely
> >> > > > > > > > that
> >> > > > > > > > >> > > > > `kafka-topics.sh` will eventually need the
> >> ability
> >> > to
> >> > > > show
> >> > > > > > the
> >> > > > > > > > id
> >> > > > > > > > >> of
> >> > > > > > > > >> > a
> >> > > > > > > > >> > > > > topic and perhaps find a topic name given an
> id.
> >> Is
> >> > > > there
> >> > > > > > any
> >> > > > > > > > >> reason
> >> > > > > > > > >> > > not
> >> > > > > > > > >> > > > to
> >> > > > > > > > >> > > > > implement that in this KIP?
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > Many thanks,
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > Tom
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > On Mon, Sep 21, 2020 at 9:54 PM Justine Olshan
> <
> >> > > > > > > > >> jols...@confluent.io
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > > > wrote:
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > > Hi all,
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > After thinking about it, I've decided to
> remove
> >> > the
> >> > > > > topic
> >> > > > > > > name
> >> > > > > > > > >> from
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > Fetch Request and Response after all. Since
> >> there
> >> > > are
> >> > > > so
> >> > > > > > > many
> >> > > > > > > > of
> >> > > > > > > > >> > > these
> >> > > > > > > > >> > > > > > requests per second, it is worth removing the
> >> > extra
> >> > > > > > > > information.
> >> > > > > > > > >> > I've
> >> > > > > > > > >> > > > > > updated the KIP to reflect this change.
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > Please let me know if there is anything else
> we
> >> > > should
> >> > > > > > > discuss
> >> > > > > > > > >> > before
> >> > > > > > > > >> > > > > > voting.
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > Thank you,
> >> > > > > > > > >> > > > > > Justine
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > On Fri, Sep 18, 2020 at 9:46 AM Justine
> Olshan
> >> <
> >> > > > > > > > >> > jols...@confluent.io
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > > > wrote:
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > > Hi Jun,
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > I see what you are saying. For now we can
> >> remove
> >> > > the
> >> > > > > > extra
> >> > > > > > > > >> > > > information.
> >> > > > > > > > >> > > > > > > I'll leave the option to add more fields to
> >> the
> >> > > file
> >> > > > > in
> >> > > > > > > the
> >> > > > > > > > >> > future.
> >> > > > > > > > >> > > > The
> >> > > > > > > > >> > > > > > KIP
> >> > > > > > > > >> > > > > > > has been updated to reflect this change.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > Thanks,
> >> > > > > > > > >> > > > > > > Justine
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > On Fri, Sep 18, 2020 at 8:46 AM Jun Rao <
> >> > > > > > j...@confluent.io
> >> > > > > > > >
> >> > > > > > > > >> > wrote:
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> Hi, Justine,
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> Thanks for the reply.
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> 13. If the log directory is the source of
> >> > truth,
> >> > > it
> >> > > > > > means
> >> > > > > > > > >> that
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > > > >> redundant info in the metadata file will
> be
> >> > > > ignored.
> >> > > > > > Then
> >> > > > > > > > the
> >> > > > > > > > >> > > > question
> >> > > > > > > > >> > > > > > is
> >> > > > > > > > >> > > > > > >> why do we need to put the redundant info
> in
> >> the
> >> > > > > > metadata
> >> > > > > > > > file
> >> > > > > > > > >> > now?
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> Thanks,
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> Jun
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> On Thu, Sep 17, 2020 at 5:07 PM Justine
> >> Olshan
> >> > <
> >> > > > > > > > >> > > > jols...@confluent.io>
> >> > > > > > > > >> > > > > > >> wrote:
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> > Hi Jun,
> >> > > > > > > > >> > > > > > >> > Thanks for the quick response!
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >> > 12. I've decided to bump up the versions
> >> on
> >> > the
> >> > > > > > > requests
> >> > > > > > > > >> and
> >> > > > > > > > >> > > > updated
> >> > > > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > KIP. I think it's good we thoroughly
> >> > discussed
> >> > > > the
> >> > > > > > > > options
> >> > > > > > > > >> > here,
> >> > > > > > > > >> > > > so
> >> > > > > > > > >> > > > > we
> >> > > > > > > > >> > > > > > >> know
> >> > > > > > > > >> > > > > > >> > we made a good choice. :)
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >> > 13. This is an interesting situation. I
> >> think
> >> > > if
> >> > > > > this
> >> > > > > > > > does
> >> > > > > > > > >> > occur
> >> > > > > > > > >> > > > we
> >> > > > > > > > >> > > > > > >> should
> >> > > > > > > > >> > > > > > >> > give a warning. I agree that it's hard
> to
> >> > know
> >> > > > the
> >> > > > > > > source
> >> > > > > > > > >> of
> >> > > > > > > > >> > > truth
> >> > > > > > > > >> > > > > for
> >> > > > > > > > >> > > > > > >> sure
> >> > > > > > > > >> > > > > > >> > since the directory or the file could be
> >> > > manually
> >> > > > > > > > >> modified. I
> >> > > > > > > > >> > > > guess
> >> > > > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > directory could be used as the source of
> >> > truth.
> >> > > > To
> >> > > > > be
> >> > > > > > > > >> honest,
> >> > > > > > > > >> > > I'm
> >> > > > > > > > >> > > > > not
> >> > > > > > > > >> > > > > > >> > really sure what happens in kafka when
> the
> >> > log
> >> > > > > > > directory
> >> > > > > > > > is
> >> > > > > > > > >> > > > renamed
> >> > > > > > > > >> > > > > > >> > manually in such a way. I'm also
> >> wondering if
> >> > > the
> >> > > > > > > > >> situation is
> >> > > > > > > > >> > > > > > >> recoverable
> >> > > > > > > > >> > > > > > >> > in this scenario.
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >> > Thanks,
> >> > > > > > > > >> > > > > > >> > Justine
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >> > On Thu, Sep 17, 2020 at 4:28 PM Jun Rao
> <
> >> > > > > > > > j...@confluent.io>
> >> > > > > > > > >> > > wrote:
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > Hi, Justine,
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > Thanks for the reply.
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > 12. I don't have a strong preference
> >> > either.
> >> > > > > > However,
> >> > > > > > > > if
> >> > > > > > > > >> we
> >> > > > > > > > >> > > need
> >> > > > > > > > >> > > > > IBP
> >> > > > > > > > >> > > > > > >> > > anyway, maybe it's easier to just bump
> >> up
> >> > the
> >> > > > > > version
> >> > > > > > > > for
> >> > > > > > > > >> > all
> >> > > > > > > > >> > > > > inter
> >> > > > > > > > >> > > > > > >> > broker
> >> > > > > > > > >> > > > > > >> > > requests and add the topic id field
> as a
> >> > > > regular
> >> > > > > > > > field. A
> >> > > > > > > > >> > > > regular
> >> > > > > > > > >> > > > > > >> field
> >> > > > > > > > >> > > > > > >> > is
> >> > > > > > > > >> > > > > > >> > > a bit more concise in wire transfer
> >> than a
> >> > > > > flexible
> >> > > > > > > > >> field.
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > 13. The confusion that I was referring
> >> to
> >> > is
> >> > > > > > between
> >> > > > > > > > the
> >> > > > > > > > >> > topic
> >> > > > > > > > >> > > > > name
> >> > > > > > > > >> > > > > > >> and
> >> > > > > > > > >> > > > > > >> > > partition number between the log dir
> and
> >> > the
> >> > > > > > metadata
> >> > > > > > > > >> file.
> >> > > > > > > > >> > > For
> >> > > > > > > > >> > > > > > >> example,
> >> > > > > > > > >> > > > > > >> > if
> >> > > > > > > > >> > > > > > >> > > the log dir is topicA-1 and the
> metadata
> >> > file
> >> > > > in
> >> > > > > it
> >> > > > > > > has
> >> > > > > > > > >> > topicB
> >> > > > > > > > >> > > > and
> >> > > > > > > > >> > > > > > >> > > partition 0 (say due to a bug or
> manual
> >> > > > > > > modification),
> >> > > > > > > > >> which
> >> > > > > > > > >> > > one
> >> > > > > > > > >> > > > > do
> >> > > > > > > > >> > > > > > we
> >> > > > > > > > >> > > > > > >> > use
> >> > > > > > > > >> > > > > > >> > > as the source of truth?
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > Jun
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > On Thu, Sep 17, 2020 at 3:43 PM
> Justine
> >> > > Olshan
> >> > > > <
> >> > > > > > > > >> > > > > > jols...@confluent.io>
> >> > > > > > > > >> > > > > > >> > > wrote:
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > > Hi Jun,
> >> > > > > > > > >> > > > > > >> > > > Thanks for the comments.
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > > > 12. I bumped the LeaderAndIsrRequest
> >> > > because
> >> > > > I
> >> > > > > > > > removed
> >> > > > > > > > >> the
> >> > > > > > > > >> > > > topic
> >> > > > > > > > >> > > > > > >> name
> >> > > > > > > > >> > > > > > >> > > field
> >> > > > > > > > >> > > > > > >> > > > in the response. It may be possible
> to
> >> > > avoid
> >> > > > > > > bumping
> >> > > > > > > > >> the
> >> > > > > > > > >> > > > version
> >> > > > > > > > >> > > > > > >> > without
> >> > > > > > > > >> > > > > > >> > > > that change, but I may be missing
> >> > > something.
> >> > > > > > > > >> > > > > > >> > > > I believe StopReplica is actually on
> >> > > version
> >> > > > 3
> >> > > > > > now,
> >> > > > > > > > but
> >> > > > > > > > >> > > > because
> >> > > > > > > > >> > > > > > >> > version 2
> >> > > > > > > > >> > > > > > >> > > > is flexible, I kept that listed as
> >> > version
> >> > > 2
> >> > > > on
> >> > > > > > the
> >> > > > > > > > KIP
> >> > > > > > > > >> > > page.
> >> > > > > > > > >> > > > > > >> However,
> >> > > > > > > > >> > > > > > >> > > you
> >> > > > > > > > >> > > > > > >> > > > may be right in that we may need to
> >> bump
> >> > > the
> >> > > > > > > version
> >> > > > > > > > on
> >> > > > > > > > >> > > > > > StopReplica
> >> > > > > > > > >> > > > > > >> to
> >> > > > > > > > >> > > > > > >> > > deal
> >> > > > > > > > >> > > > > > >> > > > with deletion differently as
> mentioned
> >> > > > above. I
> >> > > > > > > don't
> >> > > > > > > > >> know
> >> > > > > > > > >> > > if
> >> > > > > > > > >> > > > I
> >> > > > > > > > >> > > > > > >> have a
> >> > > > > > > > >> > > > > > >> > > big
> >> > > > > > > > >> > > > > > >> > > > preference over used tagged fields
> or
> >> > not.
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > > > 13. I was thinking that in the case
> >> where
> >> > > the
> >> > > > > > file
> >> > > > > > > > and
> >> > > > > > > > >> the
> >> > > > > > > > >> > > > > request
> >> > > > > > > > >> > > > > > >> > topic
> >> > > > > > > > >> > > > > > >> > > > ids don't match, it means that the
> >> > broker's
> >> > > > > > > topic/the
> >> > > > > > > > >> one
> >> > > > > > > > >> > in
> >> > > > > > > > >> > > > the
> >> > > > > > > > >> > > > > > >> file
> >> > > > > > > > >> > > > > > >> > has
> >> > > > > > > > >> > > > > > >> > > > been deleted. In that case, we would
> >> need
> >> > > to
> >> > > > > > delete
> >> > > > > > > > the
> >> > > > > > > > >> > old
> >> > > > > > > > >> > > > > topic
> >> > > > > > > > >> > > > > > >> and
> >> > > > > > > > >> > > > > > >> > > start
> >> > > > > > > > >> > > > > > >> > > > receiving the new version. If the
> >> topic
> >> > > name
> >> > > > > were
> >> > > > > > > to
> >> > > > > > > > >> > change,
> >> > > > > > > > >> > > > but
> >> > > > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > ids
> >> > > > > > > > >> > > > > > >> > > > still matched, the file would also
> >> need
> >> > to
> >> > > > > > update.
> >> > > > > > > > Am I
> >> > > > > > > > >> > > > missing
> >> > > > > > > > >> > > > > a
> >> > > > > > > > >> > > > > > >> case
> >> > > > > > > > >> > > > > > >> > > > where the file would be correct and
> >> not
> >> > the
> >> > > > > > > request?
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > > > Thanks,
> >> > > > > > > > >> > > > > > >> > > > Justine
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > > > On Thu, Sep 17, 2020 at 3:18 PM Jun
> >> Rao <
> >> > > > > > > > >> j...@confluent.io
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > > > > Hi, Justine,
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > Thanks for the reply. A couple of
> >> more
> >> > > > > comments
> >> > > > > > > > >> below.
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > 12. ListOffset and OffsetForLeader
> >> > > > currently
> >> > > > > > > don't
> >> > > > > > > > >> > support
> >> > > > > > > > >> > > > > > >> flexible
> >> > > > > > > > >> > > > > > >> > > > fields.
> >> > > > > > > > >> > > > > > >> > > > > So, we have to bump up the version
> >> > number
> >> > > > and
> >> > > > > > use
> >> > > > > > > > >> IBP at
> >> > > > > > > > >> > > > least
> >> > > > > > > > >> > > > > > for
> >> > > > > > > > >> > > > > > >> > > these
> >> > > > > > > > >> > > > > > >> > > > > two requests. Note that it seems
> >> 2.7.0
> >> > > will
> >> > > > > > > require
> >> > > > > > > > >> IBP
> >> > > > > > > > >> > > > anyway
> >> > > > > > > > >> > > > > > >> > because
> >> > > > > > > > >> > > > > > >> > > of
> >> > > > > > > > >> > > > > > >> > > > > changes in KAFKA-10435. Also, it
> >> seems
> >> > > that
> >> > > > > the
> >> > > > > > > > >> version
> >> > > > > > > > >> > > for
> >> > > > > > > > >> > > > > > >> > > > > LeaderAndIsrRequest and
> StopReplica
> >> are
> >> > > > > bumped
> >> > > > > > > even
> >> > > > > > > > >> > though
> >> > > > > > > > >> > > > we
> >> > > > > > > > >> > > > > > only
> >> > > > > > > > >> > > > > > >> > > added
> >> > > > > > > > >> > > > > > >> > > > a
> >> > > > > > > > >> > > > > > >> > > > > tagged field. But since IBP is
> >> needed
> >> > > > anyway,
> >> > > > > > we
> >> > > > > > > > may
> >> > > > > > > > >> > want
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > > > >> revisit
> >> > > > > > > > >> > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> > > > > overall tagged field choice.
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > 13. The only downside is the
> >> potential
> >> > > > > > confusion
> >> > > > > > > on
> >> > > > > > > > >> > which
> >> > > > > > > > >> > > > one
> >> > > > > > > > >> > > > > is
> >> > > > > > > > >> > > > > > >> the
> >> > > > > > > > >> > > > > > >> > > > source
> >> > > > > > > > >> > > > > > >> > > > > of truth if they don't match.
> >> Another
> >> > > > option
> >> > > > > is
> >> > > > > > > to
> >> > > > > > > > >> > include
> >> > > > > > > > >> > > > > those
> >> > > > > > > > >> > > > > > >> > fields
> >> > > > > > > > >> > > > > > >> > > > in
> >> > > > > > > > >> > > > > > >> > > > > the metadata file when we actually
> >> > change
> >> > > > the
> >> > > > > > > > >> directory
> >> > > > > > > > >> > > > > > structure.
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > Thanks,
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > Jun
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > On Thu, Sep 17, 2020 at 2:01 PM
> >> Justine
> >> > > > > Olshan
> >> > > > > > <
> >> > > > > > > > >> > > > > > >> jols...@confluent.io
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > Hello all,
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > I've thought some more about
> >> removing
> >> > > the
> >> > > > > > topic
> >> > > > > > > > >> name
> >> > > > > > > > >> > > field
> >> > > > > > > > >> > > > > > from
> >> > > > > > > > >> > > > > > >> > some
> >> > > > > > > > >> > > > > > >> > > of
> >> > > > > > > > >> > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > requests. On closer inspection
> of
> >> the
> >> > > > > > > > >> > > requests/responses,
> >> > > > > > > > >> > > > it
> >> > > > > > > > >> > > > > > >> seems
> >> > > > > > > > >> > > > > > >> > > that
> >> > > > > > > > >> > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > internal changes would be much
> >> larger
> >> > > > than
> >> > > > > I
> >> > > > > > > > >> expected.
> >> > > > > > > > >> > > > Some
> >> > > > > > > > >> > > > > > >> > protocols
> >> > > > > > > > >> > > > > > >> > > > > > involve clients, so they would
> >> > require
> >> > > > > > changes
> >> > > > > > > > too.
> >> > > > > > > > >> > I'm
> >> > > > > > > > >> > > > > > thinking
> >> > > > > > > > >> > > > > > >> > that
> >> > > > > > > > >> > > > > > >> > > > for
> >> > > > > > > > >> > > > > > >> > > > > > now, removing the topic name
> from
> >> > these
> >> > > > > > > requests
> >> > > > > > > > >> and
> >> > > > > > > > >> > > > > responses
> >> > > > > > > > >> > > > > > >> are
> >> > > > > > > > >> > > > > > >> > > out
> >> > > > > > > > >> > > > > > >> > > > of
> >> > > > > > > > >> > > > > > >> > > > > > scope.
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > I have decided to just keep the
> >> > change
> >> > > > > > > > >> > > > LeaderAndIsrResponse
> >> > > > > > > > >> > > > > to
> >> > > > > > > > >> > > > > > >> > remove
> >> > > > > > > > >> > > > > > >> > > > the
> >> > > > > > > > >> > > > > > >> > > > > > topic name, and have updated the
> >> KIP
> >> > to
> >> > > > > > reflect
> >> > > > > > > > >> this
> >> > > > > > > > >> > > > > change. I
> >> > > > > > > > >> > > > > > >> have
> >> > > > > > > > >> > > > > > >> > > > also
> >> > > > > > > > >> > > > > > >> > > > > > mentioned the other requests and
> >> > > > responses
> >> > > > > in
> >> > > > > > > > >> future
> >> > > > > > > > >> > > work.
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > I'm hoping to start the voting
> >> > process
> >> > > > > soon,
> >> > > > > > so
> >> > > > > > > > >> let me
> >> > > > > > > > >> > > > know
> >> > > > > > > > >> > > > > if
> >> > > > > > > > >> > > > > > >> > there
> >> > > > > > > > >> > > > > > >> > > is
> >> > > > > > > > >> > > > > > >> > > > > > anything else we should discuss.
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > Thank you,
> >> > > > > > > > >> > > > > > >> > > > > > Justine
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > On Tue, Sep 15, 2020 at 3:57 PM
> >> > Justine
> >> > > > > > Olshan
> >> > > > > > > <
> >> > > > > > > > >> > > > > > >> > jols...@confluent.io
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > Hello again,
> >> > > > > > > > >> > > > > > >> > > > > > > To follow up on some of the
> >> other
> >> > > > > comments:
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > 10/11) We can remove the topic
> >> name
> >> > > > from
> >> > > > > > > these
> >> > > > > > > > >> > > > > > >> > requests/responses,
> >> > > > > > > > >> > > > > > >> > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > that means we will just have
> to
> >> > make
> >> > > a
> >> > > > > few
> >> > > > > > > > >> internal
> >> > > > > > > > >> > > > > changes
> >> > > > > > > > >> > > > > > to
> >> > > > > > > > >> > > > > > >> > make
> >> > > > > > > > >> > > > > > >> > > > > > > partitions accessible by topic
> >> id
> >> > and
> >> > > > > > > > partition.
> >> > > > > > > > >> I
> >> > > > > > > > >> > can
> >> > > > > > > > >> > > > > > update
> >> > > > > > > > >> > > > > > >> the
> >> > > > > > > > >> > > > > > >> > > KIP
> >> > > > > > > > >> > > > > > >> > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > > remove them unless anyone
> thinks
> >> > they
> >> > > > > > should
> >> > > > > > > > >> stay.
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > 12) Addressed in the previous
> >> > email.
> >> > > > I've
> >> > > > > > > > updated
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > KIP
> >> > > > > > > > >> > > > > to
> >> > > > > > > > >> > > > > > >> > > include
> >> > > > > > > > >> > > > > > >> > > > > > > tagged fields for the requests
> >> and
> >> > > > > > responses.
> >> > > > > > > > >> (More
> >> > > > > > > > >> > on
> >> > > > > > > > >> > > > > that
> >> > > > > > > > >> > > > > > >> > below)
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > 13) I think part of the idea
> for
> >> > > > > including
> >> > > > > > > this
> >> > > > > > > > >> > > > > information
> >> > > > > > > > >> > > > > > >> is to
> >> > > > > > > > >> > > > > > >> > > > > prepare
> >> > > > > > > > >> > > > > > >> > > > > > > for future changes. Perhaps
> the
> >> > > > directory
> >> > > > > > > > >> structure
> >> > > > > > > > >> > > > might
> >> > > > > > > > >> > > > > > >> change
> >> > > > > > > > >> > > > > > >> > > from
> >> > > > > > > > >> > > > > > >> > > > > > > topicName_partitionNumber to
> >> > > something
> >> > > > > like
> >> > > > > > > > >> > > > > > >> > > topicID_partitionNumber.
> >> > > > > > > > >> > > > > > >> > > > > Then
> >> > > > > > > > >> > > > > > >> > > > > > > it would be useful to have the
> >> > topic
> >> > > > name
> >> > > > > > in
> >> > > > > > > > the
> >> > > > > > > > >> > file
> >> > > > > > > > >> > > > > since
> >> > > > > > > > >> > > > > > it
> >> > > > > > > > >> > > > > > >> > > would
> >> > > > > > > > >> > > > > > >> > > > > not
> >> > > > > > > > >> > > > > > >> > > > > > be
> >> > > > > > > > >> > > > > > >> > > > > > > in the directory structure.
> >> > > Supporting
> >> > > > > > topic
> >> > > > > > > > >> renames
> >> > > > > > > > >> > > > might
> >> > > > > > > > >> > > > > > be
> >> > > > > > > > >> > > > > > >> > > easier
> >> > > > > > > > >> > > > > > >> > > > if
> >> > > > > > > > >> > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > other fields are included.
> Would
> >> > > there
> >> > > > be
> >> > > > > > any
> >> > > > > > > > >> > > downsides
> >> > > > > > > > >> > > > to
> >> > > > > > > > >> > > > > > >> > > including
> >> > > > > > > > >> > > > > > >> > > > > this
> >> > > > > > > > >> > > > > > >> > > > > > > information?
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > 14)  Yes, we would need to
> copy
> >> the
> >> > > > > > partition
> >> > > > > > > > >> > metadata
> >> > > > > > > > >> > > > > file
> >> > > > > > > > >> > > > > > in
> >> > > > > > > > >> > > > > > >> > this
> >> > > > > > > > >> > > > > > >> > > > > > > process. I've updated the KIP
> to
> >> > > > include
> >> > > > > > > this.
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > 15) I believe Lucas meant v1
> >> and v2
> >> > > > here.
> >> > > > > > He
> >> > > > > > > > was
> >> > > > > > > > >> > > > referring
> >> > > > > > > > >> > > > > > to
> >> > > > > > > > >> > > > > > >> how
> >> > > > > > > > >> > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> > > > > > > requests would fall under
> >> different
> >> > > IBP
> >> > > > > and
> >> > > > > > > > meant
> >> > > > > > > > >> > that
> >> > > > > > > > >> > > > > older
> >> > > > > > > > >> > > > > > >> > > brokers
> >> > > > > > > > >> > > > > > >> > > > > > would
> >> > > > > > > > >> > > > > > >> > > > > > > have to use the older version
> of
> >> > the
> >> > > > > > request
> >> > > > > > > > and
> >> > > > > > > > >> the
> >> > > > > > > > >> > > > > > existing
> >> > > > > > > > >> > > > > > >> > topic
> >> > > > > > > > >> > > > > > >> > > > > > > deletion process. At first, it
> >> > seemed
> >> > > > > like
> >> > > > > > > > tagged
> >> > > > > > > > >> > > fields
> >> > > > > > > > >> > > > > > would
> >> > > > > > > > >> > > > > > >> > > > resolve
> >> > > > > > > > >> > > > > > >> > > > > > > the IBP issue. However, we may
> >> need
> >> > > IBP
> >> > > > > for
> >> > > > > > > > this
> >> > > > > > > > >> > > request
> >> > > > > > > > >> > > > > > after
> >> > > > > > > > >> > > > > > >> > all
> >> > > > > > > > >> > > > > > >> > > > > since
> >> > > > > > > > >> > > > > > >> > > > > > > the controller handles the
> topic
> >> > > > deletion
> >> > > > > > > > >> > differently
> >> > > > > > > > >> > > > > > >> depending
> >> > > > > > > > >> > > > > > >> > on
> >> > > > > > > > >> > > > > > >> > > > the
> >> > > > > > > > >> > > > > > >> > > > > > IBP
> >> > > > > > > > >> > > > > > >> > > > > > > version. In an older version,
> we
> >> > > can't
> >> > > > > just
> >> > > > > > > > send
> >> > > > > > > > >> a
> >> > > > > > > > >> > > > > > StopReplica
> >> > > > > > > > >> > > > > > >> > > delete
> >> > > > > > > > >> > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > topic immediately like we'd
> >> want to
> >> > > for
> >> > > > > > this
> >> > > > > > > > KIP.
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > This makes me wonder if we
> want
> >> > > tagged
> >> > > > > > fields
> >> > > > > > > > on
> >> > > > > > > > >> all
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> requests
> >> > > > > > > > >> > > > > > >> > > > after
> >> > > > > > > > >> > > > > > >> > > > > > > all. Let me know your
> thoughts!
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > Justine
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > On Tue, Sep 15, 2020 at 1:03
> PM
> >> > > Justine
> >> > > > > > > Olshan
> >> > > > > > > > <
> >> > > > > > > > >> > > > > > >> > > jols...@confluent.io
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >> Hi all,
> >> > > > > > > > >> > > > > > >> > > > > > >> Jun brought up a good point
> in
> >> his
> >> > > > last
> >> > > > > > > email
> >> > > > > > > > >> about
> >> > > > > > > > >> > > > > tagged
> >> > > > > > > > >> > > > > > >> > fields,
> >> > > > > > > > >> > > > > > >> > > > and
> >> > > > > > > > >> > > > > > >> > > > > > >> I've updated the KIP to
> reflect
> >> > that
> >> > > > the
> >> > > > > > > > >> changes to
> >> > > > > > > > >> > > > > > requests
> >> > > > > > > > >> > > > > > >> and
> >> > > > > > > > >> > > > > > >> > > > > > responses
> >> > > > > > > > >> > > > > > >> > > > > > >> will be in the form of tagged
> >> > fields
> >> > > > to
> >> > > > > > > avoid
> >> > > > > > > > >> > > changing
> >> > > > > > > > >> > > > > IBP.
> >> > > > > > > > >> > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > >> Jun: I plan on sending a
> >> followup
> >> > > > email
> >> > > > > to
> >> > > > > > > > >> address
> >> > > > > > > > >> > > some
> >> > > > > > > > >> > > > > of
> >> > > > > > > > >> > > > > > >> the
> >> > > > > > > > >> > > > > > >> > > other
> >> > > > > > > > >> > > > > > >> > > > > > >> points.
> >> > > > > > > > >> > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > >> Thanks,
> >> > > > > > > > >> > > > > > >> > > > > > >> Justine
> >> > > > > > > > >> > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > >> On Mon, Sep 14, 2020 at 4:25
> PM
> >> > Jun
> >> > > > Rao
> >> > > > > <
> >> > > > > > > > >> > > > > j...@confluent.io>
> >> > > > > > > > >> > > > > > >> > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > >>> Hi, Justine,
> >> > > > > > > > >> > > > > > >> > > > > > >>>
> >> > > > > > > > >> > > > > > >> > > > > > >>> Thanks for the updated KIP.
> A
> >> few
> >> > > > > > comments
> >> > > > > > > > >> below.
> >> > > > > > > > >> > > > > > >> > > > > > >>>
> >> > > > > > > > >> > > > > > >> > > > > > >>> 10. LeaderAndIsr Response:
> Do
> >> we
> >> > > need
> >> > > > > the
> >> > > > > > > > topic
> >> > > > > > > > >> > > name?
> >> > > > > > > > >> > > > > > >> > > > > > >>>
> >> > > > > > > > >> > > > > > >> > > > > > >>> 11. For the changed
> >> > > request/response,
> >> > > > > > other
> >> > > > > > > > >> than
> >> > > > > > > > >> > > > > > >> LeaderAndIsr,
> >> > > > > > > > >> > > > > > >> > > > > > >>> UpdateMetadata, Metadata, do
> >> we
> >> > > need
> >> > > > to
> >> > > > > > > > include
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > > topic
> >> > > > > > > > >> > > > > > >> name?
> >> > > > > > > > >> > > > > > >> > > > > > >>>
> >> > > > > > > > >> > > > > > >> > > > > > >>> 12. It seems that upgrades
> >> don't
> >> > > > > require
> >> > > > > > > IBP.
> >> > > > > > > > >> Does
> >> > > > > > > > >> > > > that
> >> > > > > > > > >> > > > > > mean
> >> > > > > > > > >> > > > > > >> > the
> >> > > > > > > > >> > > > > > >> > > > new
> >> > > > > > > > >> > > > > > >> > > > > > >>> fields
> >> > > > > > > > >> > > > > > >> > > > > > >>> in all the request/response
> >> are
> >> > > added
> >> > > > > as
> >> > > > > > > > tagged
> >> > > > > > > > >> > > fields
> >> > > > > > > > >> > > > > > >> without
> >> > > > > > > > >> > > > > > >> > > > > bumping
> >> > > > > > > > >> > > > > > >> > > > > > up
> >> > > > > > > > >> > > > > > >> > > > > > >>> the request version? It
> would
> >> be
> >> > > > useful
> >> > > > > > to
> >> > > > > > > > make
> >> > > > > > > > >> > that
> >> > > > > > > > >> > > > > > clear.
> >> > > > > > > > >> > > > > > >> > > > > > >>>
> >> > > > > > > > >> > > > > > >> > > > > > >>> 13. Partition Metadata file:
> >> Do
> >> > we
> >> > > > need
> >> > > > > > to
> >> > > > > > > > >> include
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > topic
> >> > > > > > > > >> > > > > > >> > name
> >> > > > > > > > >> > > > > > >> > > > and
> >> > > > > > > > >> > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > >>> partition id since they are
> >> > implied
> >> > > > in
> >> > > > > > the
> >> > > > > > > > >> > directory
> >> > > > > > > > >> > > > > name?
> >> > > > > > > > >> > > > > > >> > > > > > >>>
> >> > > > > > > > >> > > > > > >> > > > > > >>> 14. In the JBOD mode, we
> >> support
> >> > > > > moving a
> >> > > > > > > > >> > > partition's
> >> > > > > > > > >> > > > > data
> >> > > > > > > > >> > > > > > >> from
> >> > > > > > > > >> > > > > > >> > > one
> >> > > > > > > > >> > > > > > >> > > > > > disk
> >> > > > > > > > >> > > > > > >> > > > > > >>> to
> >> > > > > > > > >> > > > > > >> > > > > > >>> another. Will the new
> >> partition
> >> > > > > metadata
> >> > > > > > > file
> >> > > > > > > > >> be
> >> > > > > > > > >> > > > copied
> >> > > > > > > > >> > > > > > >> during
> >> > > > > > > > >> > > > > > >> > > that
> >> > > > > > > > >> > > > > > >> > > > > > >>> process?
> >> > > > > > > > >> > > > > > >> > > > > > >>>
> >> > > > > > > > >> > > > > > >> > > > > > >>> 15. The KIP says "Remove
> >> deleted
> >> > > > topics
> >> > > > > > > from
> >> > > > > > > > >> > > replicas
> >> > > > > > > > >> > > > by
> >> > > > > > > > >> > > > > > >> > sending
> >> > > > > > > > >> > > > > > >> > > > > > >>> StopReplicaRequest V2 for
> any
> >> > > topics
> >> > > > > > which
> >> > > > > > > do
> >> > > > > > > > >> not
> >> > > > > > > > >> > > > > contain
> >> > > > > > > > >> > > > > > a
> >> > > > > > > > >> > > > > > >> > topic
> >> > > > > > > > >> > > > > > >> > > > ID,
> >> > > > > > > > >> > > > > > >> > > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > >>> V3 for any topics which do
> >> > contain
> >> > > a
> >> > > > > > topic
> >> > > > > > > > >> ID.".
> >> > > > > > > > >> > > > > However,
> >> > > > > > > > >> > > > > > it
> >> > > > > > > > >> > > > > > >> > > seems
> >> > > > > > > > >> > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > >>> updated controller will
> create
> >> > all
> >> > > > > > missing
> >> > > > > > > > >> topic
> >> > > > > > > > >> > IDs
> >> > > > > > > > >> > > > > first
> >> > > > > > > > >> > > > > > >> > before
> >> > > > > > > > >> > > > > > >> > > > > doing
> >> > > > > > > > >> > > > > > >> > > > > > >>> other actions. So, is
> >> > > > > StopReplicaRequest
> >> > > > > > V2
> >> > > > > > > > >> > needed?
> >> > > > > > > > >> > > > > > >> > > > > > >>>
> >> > > > > > > > >> > > > > > >> > > > > > >>> Jun
> >> > > > > > > > >> > > > > > >> > > > > > >>>
> >> > > > > > > > >> > > > > > >> > > > > > >>> On Fri, Sep 11, 2020 at
> 10:31
> >> AM
> >> > > John
> >> > > > > > > > Roesler <
> >> > > > > > > > >> > > > > > >> > > vvcep...@apache.org
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> wrote:
> >> > > > > > > > >> > > > > > >> > > > > > >>>
> >> > > > > > > > >> > > > > > >> > > > > > >>> > Thanks, Justine!
> >> > > > > > > > >> > > > > > >> > > > > > >>> >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > Your response seems
> >> compelling
> >> > to
> >> > > > me.
> >> > > > > > > > >> > > > > > >> > > > > > >>> >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > -John
> >> > > > > > > > >> > > > > > >> > > > > > >>> >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > On Fri, 2020-09-11 at
> 10:17
> >> > > -0700,
> >> > > > > > > Justine
> >> > > > > > > > >> > Olshan
> >> > > > > > > > >> > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > Hello all,
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > Thanks for continuing
> the
> >> > > > > > discussion! I
> >> > > > > > > > >> have a
> >> > > > > > > > >> > > few
> >> > > > > > > > >> > > > > > >> > responses
> >> > > > > > > > >> > > > > > >> > > to
> >> > > > > > > > >> > > > > > >> > > > > > your
> >> > > > > > > > >> > > > > > >> > > > > > >>> > points.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > Tom: You are correct in
> >> that
> >> > > this
> >> > > > > KIP
> >> > > > > > > has
> >> > > > > > > > >> not
> >> > > > > > > > >> > > > > > mentioned
> >> > > > > > > > >> > > > > > >> the
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > DeleteTopicsRequest. I
> >> think
> >> > > that
> >> > > > > > this
> >> > > > > > > > >> would
> >> > > > > > > > >> > be
> >> > > > > > > > >> > > > out
> >> > > > > > > > >> > > > > of
> >> > > > > > > > >> > > > > > >> > scope
> >> > > > > > > > >> > > > > > >> > > > for
> >> > > > > > > > >> > > > > > >> > > > > > >>> now, but
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > may be something worth
> >> adding
> >> > > in
> >> > > > > the
> >> > > > > > > > >> future.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > John: We did consider
> >> > sequence
> >> > > > ids,
> >> > > > > > but
> >> > > > > > > > >> there
> >> > > > > > > > >> > > are
> >> > > > > > > > >> > > > a
> >> > > > > > > > >> > > > > > few
> >> > > > > > > > >> > > > > > >> > > reasons
> >> > > > > > > > >> > > > > > >> > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > >>> favor
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > UUIDs. There are several
> >> > cases
> >> > > > > where
> >> > > > > > > > topics
> >> > > > > > > > >> > from
> >> > > > > > > > >> > > > > > >> different
> >> > > > > > > > >> > > > > > >> > > > > clusters
> >> > > > > > > > >> > > > > > >> > > > > > >>> may
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > interact now and in the
> >> > future.
> >> > > > For
> >> > > > > > > > >> example,
> >> > > > > > > > >> > > > Mirror
> >> > > > > > > > >> > > > > > >> Maker 2
> >> > > > > > > > >> > > > > > >> > > may
> >> > > > > > > > >> > > > > > >> > > > > > >>> benefit
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > from being able to
> detect
> >> > when
> >> > > a
> >> > > > > > > cluster
> >> > > > > > > > >> being
> >> > > > > > > > >> > > > > > mirrored
> >> > > > > > > > >> > > > > > >> is
> >> > > > > > > > >> > > > > > >> > > > > deleted
> >> > > > > > > > >> > > > > > >> > > > > > >>> and
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > recreated and globally
> >> unique
> >> > > > > > > identifiers
> >> > > > > > > > >> > would
> >> > > > > > > > >> > > > make
> >> > > > > > > > >> > > > > > >> > > resolving
> >> > > > > > > > >> > > > > > >> > > > > > issues
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > easier than sequence IDs
> >> > which
> >> > > > may
> >> > > > > > > > collide
> >> > > > > > > > >> > > between
> >> > > > > > > > >> > > > > > >> > clusters.
> >> > > > > > > > >> > > > > > >> > > > > > KIP-405
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > (tiered storage) will
> also
> >> > > > benefit
> >> > > > > > from
> >> > > > > > > > >> > globally
> >> > > > > > > > >> > > > > > unique
> >> > > > > > > > >> > > > > > >> IDs
> >> > > > > > > > >> > > > > > >> > > as
> >> > > > > > > > >> > > > > > >> > > > > > shared
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > buckets may be used
> >> between
> >> > > > > clusters.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > Globally unique IDs
> would
> >> > also
> >> > > > make
> >> > > > > > > > >> > > functionality
> >> > > > > > > > >> > > > > like
> >> > > > > > > > >> > > > > > >> > moving
> >> > > > > > > > >> > > > > > >> > > > > > topics
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > between disparate
> clusters
> >> > > easier
> >> > > > > in
> >> > > > > > > the
> >> > > > > > > > >> > future,
> >> > > > > > > > >> > > > > > >> simplify
> >> > > > > > > > >> > > > > > >> > any
> >> > > > > > > > >> > > > > > >> > > > > > future
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > implementations of
> backups
> >> > and
> >> > > > > > > restores,
> >> > > > > > > > >> and
> >> > > > > > > > >> > > more.
> >> > > > > > > > >> > > > > In
> >> > > > > > > > >> > > > > > >> > > general,
> >> > > > > > > > >> > > > > > >> > > > > > >>> unique IDs
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > would ensure that the
> >> source
> >> > > > > cluster
> >> > > > > > > > >> topics do
> >> > > > > > > > >> > > not
> >> > > > > > > > >> > > > > > >> conflict
> >> > > > > > > > >> > > > > > >> > > > with
> >> > > > > > > > >> > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > destination cluster
> >> topics.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > If we were to use
> sequence
> >> > ids,
> >> > > > we
> >> > > > > > > would
> >> > > > > > > > >> need
> >> > > > > > > > >> > > > > > >> sufficiently
> >> > > > > > > > >> > > > > > >> > > > large
> >> > > > > > > > >> > > > > > >> > > > > > >>> cluster
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > ids to be stored with
> the
> >> > topic
> >> > > > > > > > >> identifiers or
> >> > > > > > > > >> > > we
> >> > > > > > > > >> > > > > run
> >> > > > > > > > >> > > > > > >> the
> >> > > > > > > > >> > > > > > >> > > risk
> >> > > > > > > > >> > > > > > >> > > > of
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > collisions. This will
> >> give up
> >> > > any
> >> > > > > > > > >> advantage in
> >> > > > > > > > >> > > > > > >> compactness
> >> > > > > > > > >> > > > > > >> > > that
> >> > > > > > > > >> > > > > > >> > > > > > >>> sequence
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > numbers may bring. Given
> >> > these
> >> > > > > > > > advantages I
> >> > > > > > > > >> > > think
> >> > > > > > > > >> > > > it
> >> > > > > > > > >> > > > > > >> makes
> >> > > > > > > > >> > > > > > >> > > > sense
> >> > > > > > > > >> > > > > > >> > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > >>> use
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > UUIDs.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > Gokul: This is an
> >> interesting
> >> > > > idea,
> >> > > > > > but
> >> > > > > > > > >> this
> >> > > > > > > > >> > is
> >> > > > > > > > >> > > a
> >> > > > > > > > >> > > > > > >> breaking
> >> > > > > > > > >> > > > > > >> > > > > change.
> >> > > > > > > > >> > > > > > >> > > > > > >>> Out of
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > scope for now, but maybe
> >> > worth
> >> > > > > > > discussing
> >> > > > > > > > >> in
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > > > future.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > Hope this explains some
> of
> >> > the
> >> > > > > > > decisions,
> >> > > > > > > > >> > > > > > >> > > > > > >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > Justine
> >> > > > > > > > >> > > > > > >> > > > > > >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > On Fri, Sep 11, 2020 at
> >> 8:27
> >> > AM
> >> > > > > Gokul
> >> > > > > > > > >> Ramanan
> >> > > > > > > > >> > > > > > >> Subramanian <
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > gokul24...@gmail.com>
> >> wrote:
> >> > > > > > > > >> > > > > > >> > > > > > >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > Hi.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > Thanks for the KIP.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > Have you thought about
> >> > > whether
> >> > > > it
> >> > > > > > > makes
> >> > > > > > > > >> > sense
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > > > >> support
> >> > > > > > > > >> > > > > > >> > > > > > >>> authorizing a
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > principal for a topic
> ID
> >> > > rather
> >> > > > > > than
> >> > > > > > > a
> >> > > > > > > > >> topic
> >> > > > > > > > >> > > > name
> >> > > > > > > > >> > > > > to
> >> > > > > > > > >> > > > > > >> > > achieve
> >> > > > > > > > >> > > > > > >> > > > > > >>> tighter
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > security?
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > Or is the topic ID
> >> > > > fundamentally
> >> > > > > an
> >> > > > > > > > >> internal
> >> > > > > > > > >> > > > > detail
> >> > > > > > > > >> > > > > > >> > similar
> >> > > > > > > > >> > > > > > >> > > > to
> >> > > > > > > > >> > > > > > >> > > > > > >>> epochs
> >> > > > > > > > >> > > > > > >> > > > > > >>> > used
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > in a bunch of other
> >> places
> >> > in
> >> > > > > > Kafka?
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > Thanks.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > On Fri, Sep 11, 2020
> at
> >> > 4:06
> >> > > PM
> >> > > > > > John
> >> > > > > > > > >> > Roesler <
> >> > > > > > > > >> > > > > > >> > > > > > vvcep...@apache.org>
> >> > > > > > > > >> > > > > > >> > > > > > >>> > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > Hello Justine,
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > Thanks for the KIP!
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > I happen to have
> been
> >> > > > > confronted
> >> > > > > > > > >> recently
> >> > > > > > > > >> > > with
> >> > > > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> need
> >> > > > > > > > >> > > > > > >> > > to
> >> > > > > > > > >> > > > > > >> > > > > keep
> >> > > > > > > > >> > > > > > >> > > > > > >>> > track of
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > a
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > large number of
> >> topics as
> >> > > > > > compactly
> >> > > > > > > > as
> >> > > > > > > > >> > > > > possible. I
> >> > > > > > > > >> > > > > > >> was
> >> > > > > > > > >> > > > > > >> > > > going
> >> > > > > > > > >> > > > > > >> > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > >>> come
> >> > > > > > > > >> > > > > > >> > > > > > >>> > up
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > with some way to
> >> > dictionary
> >> > > > > > encode
> >> > > > > > > > the
> >> > > > > > > > >> > topic
> >> > > > > > > > >> > > > > names
> >> > > > > > > > >> > > > > > >> as
> >> > > > > > > > >> > > > > > >> > > > > integers,
> >> > > > > > > > >> > > > > > >> > > > > > >>> but
> >> > > > > > > > >> > > > > > >> > > > > > >>> > this
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > seems much better!
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > Apologies if this
> has
> >> > been
> >> > > > > raised
> >> > > > > > > > >> before,
> >> > > > > > > > >> > > but
> >> > > > > > > > >> > > > > I’m
> >> > > > > > > > >> > > > > > >> > > wondering
> >> > > > > > > > >> > > > > > >> > > > > > >>> about the
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > choice of UUID vs
> >> > sequence
> >> > > > > number
> >> > > > > > > for
> >> > > > > > > > >> the
> >> > > > > > > > >> > > ids.
> >> > > > > > > > >> > > > > > >> > Typically,
> >> > > > > > > > >> > > > > > >> > > > > I’ve
> >> > > > > > > > >> > > > > > >> > > > > > >>> seen
> >> > > > > > > > >> > > > > > >> > > > > > >>> > UUIDs
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > in two situations:
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > 1. When processes
> >> need to
> >> > > > > > generate
> >> > > > > > > > >> > > > non-colliding
> >> > > > > > > > >> > > > > > >> > > > identifiers
> >> > > > > > > > >> > > > > > >> > > > > > >>> without
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > coordination.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > 2. When the
> identifier
> >> > > needs
> >> > > > to
> >> > > > > > be
> >> > > > > > > > >> > > > “universally
> >> > > > > > > > >> > > > > > >> > unique”;
> >> > > > > > > > >> > > > > > >> > > > > I.e.,
> >> > > > > > > > >> > > > > > >> > > > > > >>> the
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > identifier needs to
> >> > > > distinguish
> >> > > > > > the
> >> > > > > > > > >> entity
> >> > > > > > > > >> > > > from
> >> > > > > > > > >> > > > > > all
> >> > > > > > > > >> > > > > > >> > other
> >> > > > > > > > >> > > > > > >> > > > > > >>> entities
> >> > > > > > > > >> > > > > > >> > > > > > >>> > that
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > could ever exist.
> >> This is
> >> > > > > useful
> >> > > > > > in
> >> > > > > > > > >> cases
> >> > > > > > > > >> > > > where
> >> > > > > > > > >> > > > > > >> > entities
> >> > > > > > > > >> > > > > > >> > > > from
> >> > > > > > > > >> > > > > > >> > > > > > all
> >> > > > > > > > >> > > > > > >> > > > > > >>> > kinds
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > of
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > systems get mixed
> >> > together,
> >> > > > > such
> >> > > > > > as
> >> > > > > > > > >> when
> >> > > > > > > > >> > > > dumping
> >> > > > > > > > >> > > > > > >> logs
> >> > > > > > > > >> > > > > > >> > > from
> >> > > > > > > > >> > > > > > >> > > > > all
> >> > > > > > > > >> > > > > > >> > > > > > >>> > processes
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > in
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > a company into a
> >> common
> >> > > > system.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > Maybe I’m being
> >> > > > short-sighted,
> >> > > > > > but
> >> > > > > > > it
> >> > > > > > > > >> > > doesn’t
> >> > > > > > > > >> > > > > seem
> >> > > > > > > > >> > > > > > >> like
> >> > > > > > > > >> > > > > > >> > > > > either
> >> > > > > > > > >> > > > > > >> > > > > > >>> really
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > applies here. It
> seems
> >> > like
> >> > > > the
> >> > > > > > > > brokers
> >> > > > > > > > >> > > could
> >> > > > > > > > >> > > > > and
> >> > > > > > > > >> > > > > > >> would
> >> > > > > > > > >> > > > > > >> > > > > achieve
> >> > > > > > > > >> > > > > > >> > > > > > >>> > consensus
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > when creating a
> topic
> >> > > anyway,
> >> > > > > > which
> >> > > > > > > > is
> >> > > > > > > > >> all
> >> > > > > > > > >> > > > > that’s
> >> > > > > > > > >> > > > > > >> > > required
> >> > > > > > > > >> > > > > > >> > > > to
> >> > > > > > > > >> > > > > > >> > > > > > >>> > generate
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > non-colliding
> sequence
> >> > ids.
> >> > > > For
> >> > > > > > the
> >> > > > > > > > >> > second,
> >> > > > > > > > >> > > as
> >> > > > > > > > >> > > > > you
> >> > > > > > > > >> > > > > > >> > > mention,
> >> > > > > > > > >> > > > > > >> > > > > we
> >> > > > > > > > >> > > > > > >> > > > > > >>> could
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > assign
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > a UUID to the
> cluster
> >> as
> >> > a
> >> > > > > whole,
> >> > > > > > > > which
> >> > > > > > > > >> > > would
> >> > > > > > > > >> > > > > > render
> >> > > > > > > > >> > > > > > >> > any
> >> > > > > > > > >> > > > > > >> > > > > > resource
> >> > > > > > > > >> > > > > > >> > > > > > >>> > scoped
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > to
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > the broker
> universally
> >> > > unique
> >> > > > > as
> >> > > > > > > > well.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > The reason I mention
> >> this
> >> > > is
> >> > > > > > that,
> >> > > > > > > > >> > although
> >> > > > > > > > >> > > a
> >> > > > > > > > >> > > > > UUID
> >> > > > > > > > >> > > > > > >> is
> >> > > > > > > > >> > > > > > >> > way
> >> > > > > > > > >> > > > > > >> > > > > more
> >> > > > > > > > >> > > > > > >> > > > > > >>> > compact
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > than topic names,
> it’s
> >> > > still
> >> > > > 16
> >> > > > > > > > bytes.
> >> > > > > > > > >> In
> >> > > > > > > > >> > > > > > contrast,
> >> > > > > > > > >> > > > > > >> a
> >> > > > > > > > >> > > > > > >> > > > 4-byte
> >> > > > > > > > >> > > > > > >> > > > > > >>> integer
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > sequence id would
> give
> >> > us 4
> >> > > > > > billion
> >> > > > > > > > >> unique
> >> > > > > > > > >> > > > > topics
> >> > > > > > > > >> > > > > > >> per
> >> > > > > > > > >> > > > > > >> > > > > cluster,
> >> > > > > > > > >> > > > > > >> > > > > > >>> which
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > seems
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > like enough ;)
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > Considering the
> >> number of
> >> > > > > > different
> >> > > > > > > > >> times
> >> > > > > > > > >> > > > these
> >> > > > > > > > >> > > > > > >> topic
> >> > > > > > > > >> > > > > > >> > > > > > >>> identifiers are
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > sent
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > over the wire or
> >> stored
> >> > in
> >> > > > > > memory,
> >> > > > > > > it
> >> > > > > > > > >> > seems
> >> > > > > > > > >> > > > like
> >> > > > > > > > >> > > > > > it
> >> > > > > > > > >> > > > > > >> > might
> >> > > > > > > > >> > > > > > >> > > > be
> >> > > > > > > > >> > > > > > >> > > > > > >>> worth
> >> > > > > > > > >> > > > > > >> > > > > > >>> > the
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > additional 4x space
> >> > > savings.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > What do you think
> >> about
> >> > > this?
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > Thanks,
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > John
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > On Fri, Sep 11,
> 2020,
> >> at
> >> > > > 03:20,
> >> > > > > > Tom
> >> > > > > > > > >> > Bentley
> >> > > > > > > > >> > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > Hi Justine,
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > This looks like a
> >> very
> >> > > > > welcome
> >> > > > > > > > >> > > improvement.
> >> > > > > > > > >> > > > > > >> Thanks!
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > Maybe I missed it,
> >> but
> >> > > the
> >> > > > > KIP
> >> > > > > > > > >> doesn't
> >> > > > > > > > >> > > seem
> >> > > > > > > > >> > > > to
> >> > > > > > > > >> > > > > > >> > mention
> >> > > > > > > > >> > > > > > >> > > > > > changing
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > >
> DeleteTopicsRequest
> >> to
> >> > > > > identify
> >> > > > > > > the
> >> > > > > > > > >> > topic
> >> > > > > > > > >> > > > > using
> >> > > > > > > > >> > > > > > an
> >> > > > > > > > >> > > > > > >> > id.
> >> > > > > > > > >> > > > > > >> > > > > Maybe
> >> > > > > > > > >> > > > > > >> > > > > > >>> > that's out
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > of
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > scope, but
> >> > > > > DeleteTopicsRequest
> >> > > > > > is
> >> > > > > > > > not
> >> > > > > > > > >> > > listed
> >> > > > > > > > >> > > > > > among
> >> > > > > > > > >> > > > > > >> > the
> >> > > > > > > > >> > > > > > >> > > > > Future
> >> > > > > > > > >> > > > > > >> > > > > > >>> Work
> >> > > > > > > > >> > > > > > >> > > > > > >>> > APIs
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > either.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > Kind regards,
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > Tom
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > On Thu, Sep 10,
> >> 2020 at
> >> > > > 3:59
> >> > > > > PM
> >> > > > > > > > >> Satish
> >> > > > > > > > >> > > > > Duggana <
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > >
> >> satish.dugg...@gmail.com
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > Thanks
> >> Lucas/Justine
> >> > > for
> >> > > > > the
> >> > > > > > > nice
> >> > > > > > > > >> KIP.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > It has several
> >> > benefits
> >> > > > > which
> >> > > > > > > > also
> >> > > > > > > > >> > > include
> >> > > > > > > > >> > > > > > >> > > simplifying
> >> > > > > > > > >> > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > >>> topic
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > deletion process
> >> by
> >> > > > > > controller
> >> > > > > > > > and
> >> > > > > > > > >> > logs
> >> > > > > > > > >> > > > > > cleanup
> >> > > > > > > > >> > > > > > >> by
> >> > > > > > > > >> > > > > > >> > > > > brokers
> >> > > > > > > > >> > > > > > >> > > > > > in
> >> > > > > > > > >> > > > > > >> > > > > > >>> > corner
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > cases.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > Best,
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > Satish.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > On Wed, Sep 9,
> >> 2020
> >> > at
> >> > > > > 10:07
> >> > > > > > PM
> >> > > > > > > > >> > Justine
> >> > > > > > > > >> > > > > > Olshan <
> >> > > > > > > > >> > > > > > >> > > > > > >>> > jols...@confluent.io
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > Hello all,
> it's
> >> > been
> >> > > > > > almost a
> >> > > > > > > > >> year!
> >> > > > > > > > >> > > I've
> >> > > > > > > > >> > > > > > made
> >> > > > > > > > >> > > > > > >> > some
> >> > > > > > > > >> > > > > > >> > > > > > changes
> >> > > > > > > > >> > > > > > >> > > > > > >>> to
> >> > > > > > > > >> > > > > > >> > > > > > >>> > this
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > KIP
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > and hope to
> >> continue
> >> > > the
> >> > > > > > > > >> discussion.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > One of the
> main
> >> > > changes
> >> > > > > > I've
> >> > > > > > > > >> added
> >> > > > > > > > >> > is
> >> > > > > > > > >> > > > now
> >> > > > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > > > metadata
> >> > > > > > > > >> > > > > > >> > > > > > >>> response
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > will
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > include the
> topic
> >> ID
> >> > > (as
> >> > > > > > Colin
> >> > > > > > > > >> > > suggested).
> >> > > > > > > > >> > > > > > >> Clients
> >> > > > > > > > >> > > > > > >> > > can
> >> > > > > > > > >> > > > > > >> > > > > > >>> obtain the
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > topicID
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > of a given topic
> >> > > through
> >> > > > a
> >> > > > > > > > >> > > > TopicDescription.
> >> > > > > > > > >> > > > > > The
> >> > > > > > > > >> > > > > > >> > > > topicId
> >> > > > > > > > >> > > > > > >> > > > > > will
> >> > > > > > > > >> > > > > > >> > > > > > >>> > also be
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > included with
> the
> >> > > > > > > UpdateMetadata
> >> > > > > > > > >> > > request.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > Let me know
> what
> >> > you
> >> > > > all
> >> > > > > > > think.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > Thank you,
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > Justine
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > On 2019/09/13
> >> > > 16:38:26,
> >> > > > > > > "Colin
> >> > > > > > > > >> > > McCabe" <
> >> > > > > > > > >> > > > > > >> > > > > > cmcc...@apache.org
> >> > > > > > > > >> > > > > > >> > > > > > >>> >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Hi Lucas,
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Thanks for
> >> > tackling
> >> > > > > this.
> >> > > > > > > > >> Topic
> >> > > > > > > > >> > IDs
> >> > > > > > > > >> > > > > are a
> >> > > > > > > > >> > > > > > >> > great
> >> > > > > > > > >> > > > > > >> > > > > idea,
> >> > > > > > > > >> > > > > > >> > > > > > >>> and
> >> > > > > > > > >> > > > > > >> > > > > > >>> > this
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > is
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > a
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > really good
> >> writeup.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > For
> >> > > > > > > /brokers/topics/[topic],
> >> > > > > > > > >> the
> >> > > > > > > > >> > > > schema
> >> > > > > > > > >> > > > > > >> version
> >> > > > > > > > >> > > > > > >> > > > > should
> >> > > > > > > > >> > > > > > >> > > > > > be
> >> > > > > > > > >> > > > > > >> > > > > > >>> > bumped
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > to
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > version 3,
> rather
> >> > than
> >> > > 2.
> >> > > > > > > > KIP-455
> >> > > > > > > > >> > > bumped
> >> > > > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> > version
> >> > > > > > > > >> > > > > > >> > > > of
> >> > > > > > > > >> > > > > > >> > > > > > this
> >> > > > > > > > >> > > > > > >> > > > > > >>> > znode
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > to
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > 2
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > already :)
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Given that
> >> we're
> >> > > > going
> >> > > > > to
> >> > > > > > > be
> >> > > > > > > > >> > seeing
> >> > > > > > > > >> > > > > these
> >> > > > > > > > >> > > > > > >> > things
> >> > > > > > > > >> > > > > > >> > > as
> >> > > > > > > > >> > > > > > >> > > > > > >>> strings
> >> > > > > > > > >> > > > > > >> > > > > > >>> > as
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > lot
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > (in
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > logs, in
> >> ZooKeeper,
> >> > on
> >> > > > the
> >> > > > > > > > >> > command-line,
> >> > > > > > > > >> > > > > > etc.),
> >> > > > > > > > >> > > > > > >> > does
> >> > > > > > > > >> > > > > > >> > > it
> >> > > > > > > > >> > > > > > >> > > > > > make
> >> > > > > > > > >> > > > > > >> > > > > > >>> > sense to
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > use
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > base64 when
> >> > converting
> >> > > > them
> >> > > > > > to
> >> > > > > > > > >> > strings?
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Here is an
> >> > example
> >> > > of
> >> > > > > the
> >> > > > > > > hex
> >> > > > > > > > >> > > > > > >> representation:
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> >> > > > > > > > >> > 6fcb514b-b878-4c9d-95b7-8dc3a7ce6fd8
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > And here is
> an
> >> > > > example
> >> > > > > in
> >> > > > > > > > >> base64.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> >> > > > b8tRS7h4TJ2Vt43Dp85v2A
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > The base64
> >> > version
> >> > > > > saves
> >> > > > > > 15
> >> > > > > > > > >> > letters
> >> > > > > > > > >> > > > (to
> >> > > > > > > > >> > > > > be
> >> > > > > > > > >> > > > > > >> > fair,
> >> > > > > > > > >> > > > > > >> > > 4
> >> > > > > > > > >> > > > > > >> > > > of
> >> > > > > > > > >> > > > > > >> > > > > > >>> those
> >> > > > > > > > >> > > > > > >> > > > > > >>> > were
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > dashes that we
> >> could
> >> > > have
> >> > > > > > > elided
> >> > > > > > > > in
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > hex
> >> > > > > > > > >> > > > > > >> > > > > > representation.)
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Another
> thing
> >> to
> >> > > > > consider
> >> > > > > > > is
> >> > > > > > > > >> that
> >> > > > > > > > >> > we
> >> > > > > > > > >> > > > > > should
> >> > > > > > > > >> > > > > > >> > > specify
> >> > > > > > > > >> > > > > > >> > > > > > that
> >> > > > > > > > >> > > > > > >> > > > > > >>> the
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > all-zeroes UUID
> is
> >> > not
> >> > > a
> >> > > > > > valid
> >> > > > > > > > >> topic
> >> > > > > > > > >> > > UUID.
> >> > > > > > > > >> > > > > >  We
> >> > > > > > > > >> > > > > > >> > can't
> >> > > > > > > > >> > > > > > >> > > > use
> >> > > > > > > > >> > > > > > >> > > > > > >>> null
> >> > > > > > > > >> > > > > > >> > > > > > >>> > for
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > this
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > because we can't
> >> > pass a
> >> > > > > null
> >> > > > > > > UUID
> >> > > > > > > > >> over
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > RPC
> >> > > > > > > > >> > > > > > >> > > protocol
> >> > > > > > > > >> > > > > > >> > > > > > >>> (there
> >> > > > > > > > >> > > > > > >> > > > > > >>> > is no
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > special pattern
> >> for
> >> > > null,
> >> > > > > nor
> >> > > > > > > do
> >> > > > > > > > we
> >> > > > > > > > >> > want
> >> > > > > > > > >> > > > to
> >> > > > > > > > >> > > > > > >> waste
> >> > > > > > > > >> > > > > > >> > > space
> >> > > > > > > > >> > > > > > >> > > > > > >>> reserving
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > such
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > a
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > pattern.)
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Maybe I
> missed
> >> > it,
> >> > > > but
> >> > > > > > did
> >> > > > > > > > you
> >> > > > > > > > >> > > > describe
> >> > > > > > > > >> > > > > > >> > > "migration
> >> > > > > > > > >> > > > > > >> > > > > > of...
> >> > > > > > > > >> > > > > > >> > > > > > >>> > existing
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > topic[s] without
> >> > topic
> >> > > > IDs"
> >> > > > > > in
> >> > > > > > > > >> detail
> >> > > > > > > > >> > in
> >> > > > > > > > >> > > > any
> >> > > > > > > > >> > > > > > >> > section?
> >> > > > > > > > >> > > > > > >> > > > It
> >> > > > > > > > >> > > > > > >> > > > > > >>> seems
> >> > > > > > > > >> > > > > > >> > > > > > >>> > like
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > when
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > the new
> controller
> >> > > > becomes
> >> > > > > > > > active,
> >> > > > > > > > >> it
> >> > > > > > > > >> > > > should
> >> > > > > > > > >> > > > > > >> just
> >> > > > > > > > >> > > > > > >> > > > > generate
> >> > > > > > > > >> > > > > > >> > > > > > >>> random
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > UUIDs for
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > these, and write
> >> the
> >> > > > random
> >> > > > > > > UUIDs
> >> > > > > > > > >> back
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > > > >> > ZooKeeper.
> >> > > > > > > > >> > > > > > >> > > > It
> >> > > > > > > > >> > > > > > >> > > > > > >>> would be
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > good
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > spell that out.
> >> We
> >> > > > should
> >> > > > > > make
> >> > > > > > > > it
> >> > > > > > > > >> > clear
> >> > > > > > > > >> > > > > that
> >> > > > > > > > >> > > > > > >> this
> >> > > > > > > > >> > > > > > >> > > > > happens
> >> > > > > > > > >> > > > > > >> > > > > > >>> > regardless
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > of
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > the inter-broker
> >> > > protocol
> >> > > > > > > version
> >> > > > > > > > >> > (it's
> >> > > > > > > > >> > > a
> >> > > > > > > > >> > > > > > >> > compatible
> >> > > > > > > > >> > > > > > >> > > > > > change).
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> >> > > "LeaderAndIsrRequests
> >> > > > > > > > >> including an
> >> > > > > > > > >> > > > > > >> > > > is_every_partition
> >> > > > > > > > >> > > > > > >> > > > > > >>> flag"
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > seems a
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > bit wordy.  Can
> we
> >> > just
> >> > > > > call
> >> > > > > > > > these
> >> > > > > > > > >> > "full
> >> > > > > > > > >> > > > > > >> > > > > > >>> LeaderAndIsrRequests"?
> >> > > > > > > > >> > > > > > >> > > > > > >>> > Then
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > RPC field could
> be
> >> > > named
> >> > > > > > > "full".
> >> > > > > > > > >> > Also,
> >> > > > > > > > >> > > it
> >> > > > > > > > >> > > > > > would
> >> > > > > > > > >> > > > > > >> > > > probably
> >> > > > > > > > >> > > > > > >> > > > > > be
> >> > > > > > > > >> > > > > > >> > > > > > >>> > better
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > for the
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > RPC field to be
> an
> >> > enum
> >> > > > of
> >> > > > > {
> >> > > > > > > > >> > > UNSPECIFIED,
> >> > > > > > > > >> > > > > > >> > > INCREMENTAL,
> >> > > > > > > > >> > > > > > >> > > > > FULL
> >> > > > > > > > >> > > > > > >> > > > > > >>> }, so
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > that
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > we
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > can cleanly
> handle
> >> > old
> >> > > > > > versions
> >> > > > > > > > (by
> >> > > > > > > > >> > > > treating
> >> > > > > > > > >> > > > > > >> them
> >> > > > > > > > >> > > > > > >> > as
> >> > > > > > > > >> > > > > > >> > > > > > >>> UNSPECIFIED)
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > In the
> >> > > > > > LeaderAndIsrRequest
> >> > > > > > > > >> > section,
> >> > > > > > > > >> > > > you
> >> > > > > > > > >> > > > > > >> write
> >> > > > > > > > >> > > > > > >> > "A
> >> > > > > > > > >> > > > > > >> > > > > final
> >> > > > > > > > >> > > > > > >> > > > > > >>> > deletion
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > event
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > will be
> secheduled
> >> > for
> >> > > X
> >> > > > ms
> >> > > > > > > after
> >> > > > > > > > >> the
> >> > > > > > > > >> > > > > > >> > > > LeaderAndIsrRequest
> >> > > > > > > > >> > > > > > >> > > > > > was
> >> > > > > > > > >> > > > > > >> > > > > > >>> > first
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > received..."  I
> >> guess
> >> > > > the X
> >> > > > > > > was a
> >> > > > > > > > >> > > > > placeholder
> >> > > > > > > > >> > > > > > >> that
> >> > > > > > > > >> > > > > > >> > > you
> >> > > > > > > > >> > > > > > >> > > > > > >>> intended
> >> > > > > > > > >> > > > > > >> > > > > > >>> > to
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > replace
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > before posting?
> :)
> >> > In
> >> > > > any
> >> > > > > > > case,
> >> > > > > > > > >> this
> >> > > > > > > > >> > > > seems
> >> > > > > > > > >> > > > > > like
> >> > > > > > > > >> > > > > > >> > the
> >> > > > > > > > >> > > > > > >> > > > kind
> >> > > > > > > > >> > > > > > >> > > > > > of
> >> > > > > > > > >> > > > > > >> > > > > > >>> > thing
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > we'd
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > want a
> >> configuration
> >> > > for.
> >> > > > > > > Let's
> >> > > > > > > > >> > > describe
> >> > > > > > > > >> > > > > that
> >> > > > > > > > >> > > > > > >> > > > > > configuration
> >> > > > > > > > >> > > > > > >> > > > > > >>> key
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > somewhere
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > in this KIP,
> >> > including
> >> > > > what
> >> > > > > > its
> >> > > > > > > > >> > default
> >> > > > > > > > >> > > > > value
> >> > > > > > > > >> > > > > > >> is.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > We should
> >> > probably
> >> > > > also
> >> > > > > > > log a
> >> > > > > > > > >> > bunch
> >> > > > > > > > >> > > of
> >> > > > > > > > >> > > > > > >> messages
> >> > > > > > > > >> > > > > > >> > > at
> >> > > > > > > > >> > > > > > >> > > > > WARN
> >> > > > > > > > >> > > > > > >> > > > > > >>> level
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > when
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > something is
> >> > scheduled
> >> > > > for
> >> > > > > > > > >> deletion,
> >> > > > > > > > >> > as
> >> > > > > > > > >> > > > > well.
> >> > > > > > > > >> > > > > > >> > (Maybe
> >> > > > > > > > >> > > > > > >> > > > > this
> >> > > > > > > > >> > > > > > >> > > > > > >>> was
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > assumed, but
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > it would be good
> >> to
> >> > > > mention
> >> > > > > > > it).
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > I feel like
> >> there
> >> > > > are a
> >> > > > > > few
> >> > > > > > > > >> > sections
> >> > > > > > > > >> > > > > that
> >> > > > > > > > >> > > > > > >> > should
> >> > > > > > > > >> > > > > > >> > > be
> >> > > > > > > > >> > > > > > >> > > > > > >>> moved to
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > "rejected
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > alternatives."
> >> For
> >> > > > > example,
> >> > > > > > in
> >> > > > > > > > the
> >> > > > > > > > >> > > > > > DeleteTopics
> >> > > > > > > > >> > > > > > >> > > > section,
> >> > > > > > > > >> > > > > > >> > > > > > >>> since
> >> > > > > > > > >> > > > > > >> > > > > > >>> > we're
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > not
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > going to do
> >> option 1
> >> > or
> >> > > > 2,
> >> > > > > > > these
> >> > > > > > > > >> > should
> >> > > > > > > > >> > > be
> >> > > > > > > > >> > > > > > moved
> >> > > > > > > > >> > > > > > >> > into
> >> > > > > > > > >> > > > > > >> > > > > > >>> "rejected
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > alternatives,"
> >> > rather
> >> > > > than
> >> > > > > > > > >> appearing
> >> > > > > > > > >> > > > > inline.
> >> > > > > > > > >> > > > > > >> > > Another
> >> > > > > > > > >> > > > > > >> > > > > case
> >> > > > > > > > >> > > > > > >> > > > > > >>> is
> >> > > > > > > > >> > > > > > >> > > > > > >>> > the
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > "Should
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > we remove topic
> >> name
> >> > > from
> >> > > > > the
> >> > > > > > > > >> protocol
> >> > > > > > > > >> > > > where
> >> > > > > > > > >> > > > > > >> > > possible"
> >> > > > > > > > >> > > > > > >> > > > > > >>> section.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > This
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > is
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > clearly
> >> discussing a
> >> > > > design
> >> > > > > > > > >> > alternative
> >> > > > > > > > >> > > > that
> >> > > > > > > > >> > > > > > >> we're
> >> > > > > > > > >> > > > > > >> > > not
> >> > > > > > > > >> > > > > > >> > > > > > >>> proposing
> >> > > > > > > > >> > > > > > >> > > > > > >>> > to
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > implement:
> >> removing
> >> > the
> >> > > > > topic
> >> > > > > > > > name
> >> > > > > > > > >> > from
> >> > > > > > > > >> > > > > those
> >> > > > > > > > >> > > > > > >> > > > protocols.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Is it really
> >> > > > necessary
> >> > > > > to
> >> > > > > > > > have
> >> > > > > > > > >> a
> >> > > > > > > > >> > new
> >> > > > > > > > >> > > > > > >> > > > > > >>> > /admin/delete_topics_by_id
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > path
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > in ZooKeeper?
> It
> >> > seems
> >> > > > > like
> >> > > > > > we
> >> > > > > > > > >> don't
> >> > > > > > > > >> > > > really
> >> > > > > > > > >> > > > > > >> need
> >> > > > > > > > >> > > > > > >> > > this.
> >> > > > > > > > >> > > > > > >> > > > > > >>> Whenever
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > there is
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > a new
> controller,
> >> > we'll
> >> > > > > send
> >> > > > > > > out
> >> > > > > > > > >> full
> >> > > > > > > > >> > > > > > >> > > > > LeaderAndIsrRequests
> >> > > > > > > > >> > > > > > >> > > > > > >>> which
> >> > > > > > > > >> > > > > > >> > > > > > >>> > will
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > trigger the
> stale
> >> > > topics
> >> > > > to
> >> > > > > > be
> >> > > > > > > > >> cleaned
> >> > > > > > > > >> > > up.
> >> > > > > > > > >> > > > > >  The
> >> > > > > > > > >> > > > > > >> > > active
> >> > > > > > > > >> > > > > > >> > > > > > >>> > controller
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > will
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > also send the
> full
> >> > > > > > > > >> LeaderAndIsrRequest
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > > > >> brokers
> >> > > > > > > > >> > > > > > >> > > that
> >> > > > > > > > >> > > > > > >> > > > > are
> >> > > > > > > > >> > > > > > >> > > > > > >>> just
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > starting
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > up.    So we
> don't
> >> > > really
> >> > > > > > need
> >> > > > > > > > this
> >> > > > > > > > >> > kind
> >> > > > > > > > >> > > > of
> >> > > > > > > > >> > > > > > >> > two-phase
> >> > > > > > > > >> > > > > > >> > > > > > commit
> >> > > > > > > > >> > > > > > >> > > > > > >>> > (send
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > out
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > >
> >> StopReplicasRequest,
> >> > > get
> >> > > > > ACKs
> >> > > > > > > > from
> >> > > > > > > > >> all
> >> > > > > > > > >> > > > > nodes,
> >> > > > > > > > >> > > > > > >> > commit
> >> > > > > > > > >> > > > > > >> > > by
> >> > > > > > > > >> > > > > > >> > > > > > >>> removing
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > >
> >> /admin/delete_topics
> >> > > > node)
> >> > > > > > any
> >> > > > > > > > >> more.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > You mention
> >> that
> >> > > > > > > FetchRequest
> >> > > > > > > > >> will
> >> > > > > > > > >> > > now
> >> > > > > > > > >> > > > > > >> include
> >> > > > > > > > >> > > > > > >> > > UUID
> >> > > > > > > > >> > > > > > >> > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > >>> avoid
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > issues
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > where requests
> are
> >> > made
> >> > > > to
> >> > > > > > > stale
> >> > > > > > > > >> > > > partitions.
> >> > > > > > > > >> > > > > > >> > > However,
> >> > > > > > > > >> > > > > > >> > > > > > >>> adding a
> >> > > > > > > > >> > > > > > >> > > > > > >>> > UUID
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > to
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > MetadataRequest
> is
> >> > > listed
> >> > > > > as
> >> > > > > > > > future
> >> > > > > > > > >> > > work,
> >> > > > > > > > >> > > > > out
> >> > > > > > > > >> > > > > > of
> >> > > > > > > > >> > > > > > >> > > scope
> >> > > > > > > > >> > > > > > >> > > > > for
> >> > > > > > > > >> > > > > > >> > > > > > >>> this
> >> > > > > > > > >> > > > > > >> > > > > > >>> > KIP.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > How
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > will the client
> >> learn
> >> > > > what
> >> > > > > > the
> >> > > > > > > > >> topic
> >> > > > > > > > >> > > UUID
> >> > > > > > > > >> > > > > is,
> >> > > > > > > > >> > > > > > if
> >> > > > > > > > >> > > > > > >> > the
> >> > > > > > > > >> > > > > > >> > > > > > metadata
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > response
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > doesn't include
> >> that
> >> > > > > > > information?
> >> > > > > > > > >> It
> >> > > > > > > > >> > > > seems
> >> > > > > > > > >> > > > > > like
> >> > > > > > > > >> > > > > > >> > > adding
> >> > > > > > > > >> > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > >>> UUID
> >> > > > > > > > >> > > > > > >> > > > > > >>> > to
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > MetadataResponse
> >> > would
> >> > > be
> >> > > > > an
> >> > > > > > > > >> > improvement
> >> > > > > > > > >> > > > > here
> >> > > > > > > > >> > > > > > >> that
> >> > > > > > > > >> > > > > > >> > > > might
> >> > > > > > > > >> > > > > > >> > > > > > not
> >> > > > > > > > >> > > > > > >> > > > > > >>> be
> >> > > > > > > > >> > > > > > >> > > > > > >>> > too
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > hard to
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > make.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > best,
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Colin
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > On Mon, Sep
> 9,
> >> > > 2019,
> >> > > > at
> >> > > > > > > > 17:48,
> >> > > > > > > > >> > > Ryanne
> >> > > > > > > > >> > > > > > Dolan
> >> > > > > > > > >> > > > > > >> > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > Lucas,
> this
> >> > would
> >> > > > be
> >> > > > > > > great.
> >> > > > > > > > >> I've
> >> > > > > > > > >> > > run
> >> > > > > > > > >> > > > > > into
> >> > > > > > > > >> > > > > > >> > > issues
> >> > > > > > > > >> > > > > > >> > > > > with
> >> > > > > > > > >> > > > > > >> > > > > > >>> > topics
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > being
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > >
> resurrected
> >> > > > > > accidentally,
> >> > > > > > > > >> since
> >> > > > > > > > >> > a
> >> > > > > > > > >> > > > > client
> >> > > > > > > > >> > > > > > >> > cannot
> >> > > > > > > > >> > > > > > >> > > > > > easily
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > distinguish
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > between
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > a deleted
> >> topic
> >> > > > and a
> >> > > > > > new
> >> > > > > > > > >> topic
> >> > > > > > > > >> > > with
> >> > > > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> same
> >> > > > > > > > >> > > > > > >> > > > name.
> >> > > > > > > > >> > > > > > >> > > > > > I'd
> >> > > > > > > > >> > > > > > >> > > > > > >>> > need
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > the
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > ID
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > accessible
> >> from
> >> > > the
> >> > > > > > > client
> >> > > > > > > > to
> >> > > > > > > > >> > > solve
> >> > > > > > > > >> > > > > that
> >> > > > > > > > >> > > > > > >> > issue,
> >> > > > > > > > >> > > > > > >> > > > but
> >> > > > > > > > >> > > > > > >> > > > > > >>> this
> >> > > > > > > > >> > > > > > >> > > > > > >>> > is a
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > good
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > first
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > step.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > Ryanne
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > On Wed,
> Sep
> >> 4,
> >> > > 2019
> >> > > > > at
> >> > > > > > > 1:41
> >> > > > > > > > >> PM
> >> > > > > > > > >> > > Lucas
> >> > > > > > > > >> > > > > > >> > > Bradstreet <
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > lu...@confluent.io>
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Hi all,
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > I would
> >> like
> >> > to
> >> > > > > kick
> >> > > > > > > off
> >> > > > > > > > >> > > > discussion
> >> > > > > > > > >> > > > > of
> >> > > > > > > > >> > > > > > >> > > KIP-516,
> >> > > > > > > > >> > > > > > >> > > > > an
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > implementation
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > of topic
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > IDs for
> >> > Kafka.
> >> > > > > Topic
> >> > > > > > > IDs
> >> > > > > > > > >> aim
> >> > > > > > > > >> > to
> >> > > > > > > > >> > > > > solve
> >> > > > > > > > >> > > > > > >> topic
> >> > > > > > > > >> > > > > > >> > > > > > >>> uniqueness
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > problems in
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > Kafka,
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > where
> >> > referring
> >> > > > to
> >> > > > > a
> >> > > > > > > > topic
> >> > > > > > > > >> by
> >> > > > > > > > >> > > name
> >> > > > > > > > >> > > > > > >> alone is
> >> > > > > > > > >> > > > > > >> > > > > > >>> insufficient.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > Such
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > cases
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > include
> >> when
> >> > a
> >> > > > > topic
> >> > > > > > > has
> >> > > > > > > > >> been
> >> > > > > > > > >> > > > > deleted
> >> > > > > > > > >> > > > > > >> and
> >> > > > > > > > >> > > > > > >> > > > > recreated
> >> > > > > > > > >> > > > > > >> > > > > > >>> with
> >> > > > > > > > >> > > > > > >> > > > > > >>> > the
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > same
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > name.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Unique
> >> > > > identifiers
> >> > > > > > will
> >> > > > > > > > >> help
> >> > > > > > > > >> > > > > simplify
> >> > > > > > > > >> > > > > > >> and
> >> > > > > > > > >> > > > > > >> > > > improve
> >> > > > > > > > >> > > > > > >> > > > > > >>> Kafka's
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > topic
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > deletion
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > process,
> >> as
> >> > > well
> >> > > > as
> >> > > > > > > > prevent
> >> > > > > > > > >> > > cases
> >> > > > > > > > >> > > > > > where
> >> > > > > > > > >> > > > > > >> > > brokers
> >> > > > > > > > >> > > > > > >> > > > > may
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > incorrectly
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > interact
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > with
> stale
> >> > > > versions
> >> > > > > > of
> >> > > > > > > > >> topics.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> >
> >> > > > > > > > >> > > > > > >> > > > > > >>>
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Looking
> >> > forward
> >> > > > to
> >> > > > > > your
> >> > > > > > > > >> > > thoughts.
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Lucas
> >> > > > > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >>> >
> >> > > > > > > > >> > > > > > >> > > > > > >>> >
> >> > > > > > > > >> > > > > > >> > > > > > >>>
> >> > > > > > > > >> > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>