Thanks Justine. Responses below:

> 1. Yes, the directory will still be based on the topic names.
LeaderAndIsrRequest is one of the few requests that will still contain the
topic name. So I think we have this covered. Sorry for confusion.

Ah, you're right. My eyes passed right over the field.

> 2. Part of the usage of the file is to have persistent storage of the
topic
ID and use it to compare with the ID supplied in the LeaderAndIsr Request.
There is some discussion in the KIP about changes to the directory
structure, but I believe directory changes were considered to be out of
scope when the KIP was written.

Yeah, I was hoping to get a better understanding of why it was taken out of
scope. Perhaps Lucas Bradstreet might have more insight about the decision.
Basically my point is that we have to create additional infrastructure here
to support the name/id mapping, so I wanted to understand if we just
consider this a sort of tech debt. It would be useful to cover how we
handle the case when this file gets corrupted. Seems like we just have to
trust that it matches whatever the controller tells us and rewrite it?

> 3. I think this is a good point, but I again I wonder about the scope of
the KIP. Most of the changes mentioned in the KIP are for supporting topic
deletion and I believe that is why the produce request was listed under
future work.

That's fair. I brought it up since `Fetch` is already included. If we've
got `Metadata` and `Fetch`, seems we may as well do `Produce` and save an
extra kip. No strong objection though if you want to leave it out.


-Jason


On Thu, Sep 24, 2020 at 3:26 PM Justine Olshan <jols...@confluent.io> wrote:

> Hi Jason,
>
> Thanks for your comments.
>
> 1. Yes, the directory will still be based on the topic names.
> LeaderAndIsrRequest is one of the few requests that will still contain the
> topic name. So I think we have this covered. Sorry for confusion.
>
> 2. Part of the usage of the file is to have persistent storage of the topic
> ID and use it to compare with the ID supplied in the LeaderAndIsr Request.
> There is some discussion in the KIP about changes to the directory
> structure, but I believe directory changes were considered to be out of
> scope when the KIP was written.
>
> 3. I think this is a good point, but I again I wonder about the scope of
> the KIP. Most of the changes mentioned in the KIP are for supporting topic
> deletion and I believe that is why the produce request was listed under
> future work.
>
> 4. This sounds like it might be a good solution, but I will need to discuss
> more with KIP-500 folks to get the details right.
>
> Thanks,
> Justine
>
> On Thu, Sep 24, 2020 at 12:30 PM Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Hi Justine,
> >
> > Thanks for picking up this work. I have a few questions/comments:
> >
> > 1. It sounds like the directory structure is still going to be based on
> > topic names. Do I have that right? One complication is that the
> > LeaderAndIsr request does not include the topic name any longer. This
> means
> > that a replica must wait for the UpdateMetadata request to arrive with
> the
> > topic name mapping before it can create the directory. I wonder if it
> would
> > be simpler to avoid assumptions on the ordering of UpdateMetadata and let
> > LeaderAndIsr include the topic name as well. Feels like we are not saving
> > that much by excluding it.
> >
> > 2. On a related note, it seems that the reason we have the partition
> > metadata file is because we are not changing the directory structure. We
> > need it so that we remember which directories map to which topic id. I
> > think it would be useful to add some detail about why changing the
> > directory layout was rejected. Basically trying to understand how much
> > effort we are saving by skipping this step if we have to implement this
> > additional bookkeeping. One concern I have for example is that the
> > partition metadata file gets corrupt and then all bets are off as far as
> > topic consistency.
> >
> > 3. Is it worth adding support now for topic ids in the `Produce` request
> > now? Overhead is mentioned as one of the motivations and the best APIs to
> > get savings from are `Produce` and `Fetch`.
> >
> > 4. When it comes to bootstrapping the metadata topic with respect to
> > KIP-500, one suggestion would be to reserve a sentinel topic ID which can
> > be used initially by new brokers. When the first controller is elected,
> it
> > can assign a topicId to the metadata topic. This is a bit similar to how
> we
> > were planning to generate the clusterId.
> >
> > Thanks,
> > Jason
> >
> >
> > On Thu, Sep 24, 2020 at 11:10 AM Jun Rao <j...@confluent.io> wrote:
> >
> > > Hi, Justine,
> > >
> > > Thanks for the updated KIP. A few more comments below.
> > >
> > > 30.  {"name": "id", "type": "string", "doc": "version id"}}: The doc
> > should
> > > say UUID.
> > >
> > > 31. LeaderAndIsrResponse v5 and StopReplicaResponse v4 : It seems there
> > is
> > > no need to add topic_id at partitions level.
> > >
> > > 32. Regarding partition metadata file. Perhaps the key can be a single
> > > word, sth like the following.
> > > version: 0
> > > topic_id: 46bdb63f-9e8d-4a38-bf7b-ee4eb2a794e4
> > >
> > > 33. Another tricky thing that I realized is how to support the metadata
> > > topic introduced in KIP-595. It's a bit tricky to assign a UUID to the
> > > metadata topic since we have a chicken-and-egg problem. The controller
> > > needs to persist the UUID in the metadata topic in order to assign one
> > > successfully, but the metadata topic is needed to elect a controller
> > > (KIP-631). So, this probably needs a bit more thought.
> > >
> > > Jun
> > >
> > > On Thu, Sep 24, 2020 at 3:04 AM Ismael Juma <ism...@juma.me.uk> wrote:
> > >
> > > > Also, can we provide more details on how the Partition Metadata file
> > will
> > > > be used?
> > > >
> > > > Ismael
> > > >
> > > > On Thu, Sep 24, 2020 at 3:01 AM Ismael Juma <ism...@juma.me.uk>
> wrote:
> > > >
> > > > > Hi Justine,
> > > > >
> > > > > I think we need to update the "Rejected Alternatives" section to
> take
> > > > into
> > > > > account that the proposal now removes the topic name from the fetch
> > > > > request. Also, if we are removing it from the Fetch request, does
> it
> > > make
> > > > > sense not to remove it from similar requests like
> ListOffsetRequest?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Thu, Sep 24, 2020 at 2:46 AM David Jacot <dja...@confluent.io>
> > > wrote:
> > > > >
> > > > >> Hi Justine,
> > > > >>
> > > > >> Thanks for the KIP. I finally had time to read it :). It is a
> great
> > > > >> improvement.
> > > > >>
> > > > >> I have a few comments/questions:
> > > > >>
> > > > >> 1. It seems that the schema of the StopReplicaRequest is slightly
> > > > >> outdated.
> > > > >> We
> > > > >> did some changes as part of KIP-570. V3 is already organized by
> > > topics.
> > > > >>
> > > > >> 2. I just want to make sure that I understand the reconciliation
> > > > >> logic correctly. When
> > > > >> an "INCREMENTAL" LeaderAndIsr Request is received, the broker will
> > > also
> > > > >> reconcile
> > > > >> when the local uuid does not match the uuid in the request, right?
> > In
> > > > this
> > > > >> case, the
> > > > >> local log is staged for deletion.
> > > > >>
> > > > >> 3. In the documentation of the `delete.stale.topic.delay.ms`
> > config,
> > > it
> > > > >> says "When a
> > > > >> FULL LeaderAndIsrRequest is received..." but I suppose it applies
> to
> > > > both
> > > > >> types.
> > > > >>
> > > > >> Best,
> > > > >> David
> > > > >>
> > > > >> On Thu, Sep 24, 2020 at 1:40 AM Justine Olshan <
> > jols...@confluent.io>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Jun,
> > > > >> >
> > > > >> > Thanks for the comments. I apologize for some of the typos and
> > > > >> confusion.
> > > > >> > I’ve updated the KIP to fix some of the issues you mentioned.
> > > > >> >
> > > > >> > 20.2 I’ve changed the type to String.
> > > > >> > 20.1/3 I’ve updated the TopicZNode to fix formatting and reflect
> > the
> > > > >> latest
> > > > >> > version before this change.
> > > > >> >
> > > > >> > 21. You are correct and I’ve removed this line. I’ve also added
> a
> > > line
> > > > >> > mentioning an IBP bump is necessary for migration
> > > > >> >
> > > > >> > 22. I think the wording was unclear but your summary is what was
> > > > >> intended
> > > > >> > by this line. I’ve updated to make this point more clear.
> “Remove
> > > > >> deleted
> > > > >> > topics from replicas by sending StopReplicaRequest V3 before the
> > IBP
> > > > >> bump
> > > > >> > using the old logic, and using V4 and the new logic with topic
> IDs
> > > > after
> > > > >> > the IBP bump.”
> > > > >> >
> > > > >> > 23. I’ve removed the unspecified type from the KIP and mention
> > that
> > > > IBP
> > > > >> > will be used to handle this request. “IBP will be used to
> > determine
> > > > >> whether
> > > > >> > this new form of the request will be used. For older requests,
> we
> > > will
> > > > >> > ignore this field and default to previous behavior.”
> > > > >> >
> > > > >> > 24. I’ve fixed this typo.
> > > > >> >
> > > > >> > 25. I've added a topics at a higher level for
> LeaderAndIsrResponse
> > > v5,
> > > > >> > StopReplicaResponse v4. I've also changed StopReplicaRequest v4
> to
> > > > have
> > > > >> > topics at a higher level.
> > > > >> >
> > > > >> > 26. I’ve updated forgotten_topics_data--added the topic ID and
> > > removed
> > > > >> the
> > > > >> > topic name
> > > > >> >
> > > > >> > 27. I’ve decided on plain text, and I’ve added an example.
> > > > >> >
> > > > >> > 28. I’ve added this idea to future work.
> > > > >> >
> > > > >> > Thanks again for taking a look,
> > > > >> >
> > > > >> > Justine
> > > > >> >
> > > > >> > On Wed, Sep 23, 2020 at 10:28 AM Jun Rao <j...@confluent.io>
> > wrote:
> > > > >> >
> > > > >> > > Hi, Justine,
> > > > >> > >
> > > > >> > > Thanks for the response. Made another pass. A few more
> comments
> > > > below.
> > > > >> > >
> > > > >> > > 20. znode schema:
> > > > >> > > 20.1 It seems that {"name": "version", "type": "int", "id":
> > > "UUID",
> > > > >> > "doc":
> > > > >> > > "version id"} should be {"name": "version", "type": "int"},
> > > {"name":
> > > > >> > "id",
> > > > >> > > "type": "UUID", "doc": "version id"}.
> > > > >> > > 20.2 The znode format is JSON which doesn't have UUID type. So
> > the
> > > > >> type
> > > > >> > > probably should be string?
> > > > >> > > 20.3 Also, the existing format used seems outdated. It should
> > have
> > > > the
> > > > >> > > following format.
> > > > >> > >     Json.encodeAsBytes(Map(
> > > > >> > >       "version" -> 2,
> > > > >> > >       "partitions" -> replicaAssignmentJson.asJava,
> > > > >> > >       "adding_replicas" ->
> addingReplicasAssignmentJson.asJava,
> > > > >> > >       "removing_replicas" ->
> > removingReplicasAssignmentJson.asJava
> > > > >> > >     ).asJava)
> > > > >> > >   }
> > > > >> > >
> > > > >> > > 21. Migration: The KIP says "The migration process can take
> > place
> > > > >> without
> > > > >> > > an inter-broker protocol bump, as the format stored in
> > > > >> > > /brokers/topics/[topic] will be compatible with older broker
> > > > >> versions."
> > > > >> > > However, since we are bumping up the version of inter-broker
> > > > >> requests, it
> > > > >> > > seems that we need to use IBP for migration.
> > > > >> > >
> > > > >> > > 22. The KIP says "Remove deleted topics from replicas by
> sending
> > > > >> > > StopReplicaRequest V3 for any topics which do not contain a
> > topic
> > > > ID,
> > > > >> and
> > > > >> > > V4 for any topics which do contain a topic ID.". However, if
> we
> > > use
> > > > >> IBP,
> > > > >> > it
> > > > >> > > seems that the controller will either send StopReplicaRequest
> V3
> > > > >> > > or StopReplicaRequest V4, but never mixed V3 and V4 for
> > different
> > > > >> topics.
> > > > >> > > Basically, before the IBP bump, V3 will be used. After the IBP
> > > bump,
> > > > >> > > topicId will be created and V4 will be used.
> > > > >> > >
> > > > >> > > 23. Given that we depend on IBP, do we still need "0
> > UNSPECIFIED"
> > > > >> > > in LeaderAndIsr?
> > > > >> > >
> > > > >> > > 24. LeaderAndIsrResponse v5 : It still has the topic field.
> > > > >> > >
> > > > >> > > 25. LeaderAndIsrResponse v5, StopReplicaResponse v4: Could we
> > use
> > > > this
> > > > >> > > opportunity to organize the response in 2 levels, first by
> > topic,
> > > > >> then by
> > > > >> > > partition, as most other requests/responses?
> > > > >> > >
> > > > >> > > 26. FetchRequest v13 : Should forgotten_topics_data use
> topicId
> > > too?
> > > > >> > >
> > > > >> > > 27. "This file can either be plain text (key/value pairs) or
> > > JSON."
> > > > >> Have
> > > > >> > we
> > > > >> > > decided which one to use? Also, it would be helpful to provide
> > an
> > > > >> > example.
> > > > >> > >
> > > > >> > > 28. Future improvement: Another future improvement opportunity
> > is
> > > to
> > > > >> use
> > > > >> > > topicId in GroupMetadataManager.offsetCommitKey in the
> > > offset_commit
> > > > >> > topic.
> > > > >> > > This may save some space.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > >
> > > > >> > > Jun
> > > > >> > >
> > > > >> > > On Wed, Sep 23, 2020 at 8:50 AM Justine Olshan <
> > > > jols...@confluent.io>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Hi Tom,
> > > > >> > > >
> > > > >> > > > Thanks for the comment. I think this is a really good idea
> and
> > > it
> > > > >> has
> > > > >> > > been
> > > > >> > > > added to the KIP under the newly added tooling section.
> > > > >> > > >
> > > > >> > > > Thanks again,
> > > > >> > > > Justine
> > > > >> > > >
> > > > >> > > > On Wed, Sep 23, 2020 at 3:17 AM Tom Bentley <
> > > tbent...@redhat.com>
> > > > >> > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Justine,
> > > > >> > > > >
> > > > >> > > > > I know you started the vote thread, but on re-reading the
> > KIP
> > > I
> > > > >> > noticed
> > > > >> > > > > that although the topic id is included in the
> > MetadataResponse
> > > > >> it's
> > > > >> > not
> > > > >> > > > > surfaced in the output from `kafka-topics.sh --describe`.
> > > Maybe
> > > > >> that
> > > > >> > > was
> > > > >> > > > > intentional because ids are intentionally not really
> > something
> > > > the
> > > > >> > user
> > > > >> > > > > should care deeply about, but it would also make life
> harder
> > > for
> > > > >> > anyone
> > > > >> > > > > debugging Kafka and this would likely get worse the more
> > topic
> > > > ids
> > > > >> > got
> > > > >> > > > > rolled out across the protocols, clients etc. It seems
> > likely
> > > > that
> > > > >> > > > > `kafka-topics.sh` will eventually need the ability to show
> > the
> > > > id
> > > > >> of
> > > > >> > a
> > > > >> > > > > topic and perhaps find a topic name given an id. Is there
> > any
> > > > >> reason
> > > > >> > > not
> > > > >> > > > to
> > > > >> > > > > implement that in this KIP?
> > > > >> > > > >
> > > > >> > > > > Many thanks,
> > > > >> > > > >
> > > > >> > > > > Tom
> > > > >> > > > >
> > > > >> > > > > On Mon, Sep 21, 2020 at 9:54 PM Justine Olshan <
> > > > >> jols...@confluent.io
> > > > >> > >
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Hi all,
> > > > >> > > > > >
> > > > >> > > > > > After thinking about it, I've decided to remove the
> topic
> > > name
> > > > >> from
> > > > >> > > the
> > > > >> > > > > > Fetch Request and Response after all. Since there are so
> > > many
> > > > of
> > > > >> > > these
> > > > >> > > > > > requests per second, it is worth removing the extra
> > > > information.
> > > > >> > I've
> > > > >> > > > > > updated the KIP to reflect this change.
> > > > >> > > > > >
> > > > >> > > > > > Please let me know if there is anything else we should
> > > discuss
> > > > >> > before
> > > > >> > > > > > voting.
> > > > >> > > > > >
> > > > >> > > > > > Thank you,
> > > > >> > > > > > Justine
> > > > >> > > > > >
> > > > >> > > > > > On Fri, Sep 18, 2020 at 9:46 AM Justine Olshan <
> > > > >> > jols...@confluent.io
> > > > >> > > >
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Hi Jun,
> > > > >> > > > > > >
> > > > >> > > > > > > I see what you are saying. For now we can remove the
> > extra
> > > > >> > > > information.
> > > > >> > > > > > > I'll leave the option to add more fields to the file
> in
> > > the
> > > > >> > future.
> > > > >> > > > The
> > > > >> > > > > > KIP
> > > > >> > > > > > > has been updated to reflect this change.
> > > > >> > > > > > >
> > > > >> > > > > > > Thanks,
> > > > >> > > > > > > Justine
> > > > >> > > > > > >
> > > > >> > > > > > > On Fri, Sep 18, 2020 at 8:46 AM Jun Rao <
> > j...@confluent.io
> > > >
> > > > >> > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > >> Hi, Justine,
> > > > >> > > > > > >>
> > > > >> > > > > > >> Thanks for the reply.
> > > > >> > > > > > >>
> > > > >> > > > > > >> 13. If the log directory is the source of truth, it
> > means
> > > > >> that
> > > > >> > the
> > > > >> > > > > > >> redundant info in the metadata file will be ignored.
> > Then
> > > > the
> > > > >> > > > question
> > > > >> > > > > > is
> > > > >> > > > > > >> why do we need to put the redundant info in the
> > metadata
> > > > file
> > > > >> > now?
> > > > >> > > > > > >>
> > > > >> > > > > > >> Thanks,
> > > > >> > > > > > >>
> > > > >> > > > > > >> Jun
> > > > >> > > > > > >>
> > > > >> > > > > > >> On Thu, Sep 17, 2020 at 5:07 PM Justine Olshan <
> > > > >> > > > jols...@confluent.io>
> > > > >> > > > > > >> wrote:
> > > > >> > > > > > >>
> > > > >> > > > > > >> > Hi Jun,
> > > > >> > > > > > >> > Thanks for the quick response!
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > 12. I've decided to bump up the versions on the
> > > requests
> > > > >> and
> > > > >> > > > updated
> > > > >> > > > > > the
> > > > >> > > > > > >> > KIP. I think it's good we thoroughly discussed the
> > > > options
> > > > >> > here,
> > > > >> > > > so
> > > > >> > > > > we
> > > > >> > > > > > >> know
> > > > >> > > > > > >> > we made a good choice. :)
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > 13. This is an interesting situation. I think if
> this
> > > > does
> > > > >> > occur
> > > > >> > > > we
> > > > >> > > > > > >> should
> > > > >> > > > > > >> > give a warning. I agree that it's hard to know the
> > > source
> > > > >> of
> > > > >> > > truth
> > > > >> > > > > for
> > > > >> > > > > > >> sure
> > > > >> > > > > > >> > since the directory or the file could be manually
> > > > >> modified. I
> > > > >> > > > guess
> > > > >> > > > > > the
> > > > >> > > > > > >> > directory could be used as the source of truth. To
> be
> > > > >> honest,
> > > > >> > > I'm
> > > > >> > > > > not
> > > > >> > > > > > >> > really sure what happens in kafka when the log
> > > directory
> > > > is
> > > > >> > > > renamed
> > > > >> > > > > > >> > manually in such a way. I'm also wondering if the
> > > > >> situation is
> > > > >> > > > > > >> recoverable
> > > > >> > > > > > >> > in this scenario.
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > Thanks,
> > > > >> > > > > > >> > Justine
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > On Thu, Sep 17, 2020 at 4:28 PM Jun Rao <
> > > > j...@confluent.io>
> > > > >> > > wrote:
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > > Hi, Justine,
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > Thanks for the reply.
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > 12. I don't have a strong preference either.
> > However,
> > > > if
> > > > >> we
> > > > >> > > need
> > > > >> > > > > IBP
> > > > >> > > > > > >> > > anyway, maybe it's easier to just bump up the
> > version
> > > > for
> > > > >> > all
> > > > >> > > > > inter
> > > > >> > > > > > >> > broker
> > > > >> > > > > > >> > > requests and add the topic id field as a regular
> > > > field. A
> > > > >> > > > regular
> > > > >> > > > > > >> field
> > > > >> > > > > > >> > is
> > > > >> > > > > > >> > > a bit more concise in wire transfer than a
> flexible
> > > > >> field.
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > 13. The confusion that I was referring to is
> > between
> > > > the
> > > > >> > topic
> > > > >> > > > > name
> > > > >> > > > > > >> and
> > > > >> > > > > > >> > > partition number between the log dir and the
> > metadata
> > > > >> file.
> > > > >> > > For
> > > > >> > > > > > >> example,
> > > > >> > > > > > >> > if
> > > > >> > > > > > >> > > the log dir is topicA-1 and the metadata file in
> it
> > > has
> > > > >> > topicB
> > > > >> > > > and
> > > > >> > > > > > >> > > partition 0 (say due to a bug or manual
> > > modification),
> > > > >> which
> > > > >> > > one
> > > > >> > > > > do
> > > > >> > > > > > we
> > > > >> > > > > > >> > use
> > > > >> > > > > > >> > > as the source of truth?
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > Jun
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > On Thu, Sep 17, 2020 at 3:43 PM Justine Olshan <
> > > > >> > > > > > jols...@confluent.io>
> > > > >> > > > > > >> > > wrote:
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > > Hi Jun,
> > > > >> > > > > > >> > > > Thanks for the comments.
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > 12. I bumped the LeaderAndIsrRequest because I
> > > > removed
> > > > >> the
> > > > >> > > > topic
> > > > >> > > > > > >> name
> > > > >> > > > > > >> > > field
> > > > >> > > > > > >> > > > in the response. It may be possible to avoid
> > > bumping
> > > > >> the
> > > > >> > > > version
> > > > >> > > > > > >> > without
> > > > >> > > > > > >> > > > that change, but I may be missing something.
> > > > >> > > > > > >> > > > I believe StopReplica is actually on version 3
> > now,
> > > > but
> > > > >> > > > because
> > > > >> > > > > > >> > version 2
> > > > >> > > > > > >> > > > is flexible, I kept that listed as version 2 on
> > the
> > > > KIP
> > > > >> > > page.
> > > > >> > > > > > >> However,
> > > > >> > > > > > >> > > you
> > > > >> > > > > > >> > > > may be right in that we may need to bump the
> > > version
> > > > on
> > > > >> > > > > > StopReplica
> > > > >> > > > > > >> to
> > > > >> > > > > > >> > > deal
> > > > >> > > > > > >> > > > with deletion differently as mentioned above. I
> > > don't
> > > > >> know
> > > > >> > > if
> > > > >> > > > I
> > > > >> > > > > > >> have a
> > > > >> > > > > > >> > > big
> > > > >> > > > > > >> > > > preference over used tagged fields or not.
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > 13. I was thinking that in the case where the
> > file
> > > > and
> > > > >> the
> > > > >> > > > > request
> > > > >> > > > > > >> > topic
> > > > >> > > > > > >> > > > ids don't match, it means that the broker's
> > > topic/the
> > > > >> one
> > > > >> > in
> > > > >> > > > the
> > > > >> > > > > > >> file
> > > > >> > > > > > >> > has
> > > > >> > > > > > >> > > > been deleted. In that case, we would need to
> > delete
> > > > the
> > > > >> > old
> > > > >> > > > > topic
> > > > >> > > > > > >> and
> > > > >> > > > > > >> > > start
> > > > >> > > > > > >> > > > receiving the new version. If the topic name
> were
> > > to
> > > > >> > change,
> > > > >> > > > but
> > > > >> > > > > > the
> > > > >> > > > > > >> > ids
> > > > >> > > > > > >> > > > still matched, the file would also need to
> > update.
> > > > Am I
> > > > >> > > > missing
> > > > >> > > > > a
> > > > >> > > > > > >> case
> > > > >> > > > > > >> > > > where the file would be correct and not the
> > > request?
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > Thanks,
> > > > >> > > > > > >> > > > Justine
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > On Thu, Sep 17, 2020 at 3:18 PM Jun Rao <
> > > > >> j...@confluent.io
> > > > >> > >
> > > > >> > > > > wrote:
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > > Hi, Justine,
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > Thanks for the reply. A couple of more
> comments
> > > > >> below.
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > 12. ListOffset and OffsetForLeader currently
> > > don't
> > > > >> > support
> > > > >> > > > > > >> flexible
> > > > >> > > > > > >> > > > fields.
> > > > >> > > > > > >> > > > > So, we have to bump up the version number and
> > use
> > > > >> IBP at
> > > > >> > > > least
> > > > >> > > > > > for
> > > > >> > > > > > >> > > these
> > > > >> > > > > > >> > > > > two requests. Note that it seems 2.7.0 will
> > > require
> > > > >> IBP
> > > > >> > > > anyway
> > > > >> > > > > > >> > because
> > > > >> > > > > > >> > > of
> > > > >> > > > > > >> > > > > changes in KAFKA-10435. Also, it seems that
> the
> > > > >> version
> > > > >> > > for
> > > > >> > > > > > >> > > > > LeaderAndIsrRequest and StopReplica are
> bumped
> > > even
> > > > >> > though
> > > > >> > > > we
> > > > >> > > > > > only
> > > > >> > > > > > >> > > added
> > > > >> > > > > > >> > > > a
> > > > >> > > > > > >> > > > > tagged field. But since IBP is needed anyway,
> > we
> > > > may
> > > > >> > want
> > > > >> > > to
> > > > >> > > > > > >> revisit
> > > > >> > > > > > >> > > the
> > > > >> > > > > > >> > > > > overall tagged field choice.
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > 13. The only downside is the potential
> > confusion
> > > on
> > > > >> > which
> > > > >> > > > one
> > > > >> > > > > is
> > > > >> > > > > > >> the
> > > > >> > > > > > >> > > > source
> > > > >> > > > > > >> > > > > of truth if they don't match. Another option
> is
> > > to
> > > > >> > include
> > > > >> > > > > those
> > > > >> > > > > > >> > fields
> > > > >> > > > > > >> > > > in
> > > > >> > > > > > >> > > > > the metadata file when we actually change the
> > > > >> directory
> > > > >> > > > > > structure.
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > Thanks,
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > Jun
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > On Thu, Sep 17, 2020 at 2:01 PM Justine
> Olshan
> > <
> > > > >> > > > > > >> jols...@confluent.io
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > > > wrote:
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > > Hello all,
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > > > I've thought some more about removing the
> > topic
> > > > >> name
> > > > >> > > field
> > > > >> > > > > > from
> > > > >> > > > > > >> > some
> > > > >> > > > > > >> > > of
> > > > >> > > > > > >> > > > > the
> > > > >> > > > > > >> > > > > > requests. On closer inspection of the
> > > > >> > > requests/responses,
> > > > >> > > > it
> > > > >> > > > > > >> seems
> > > > >> > > > > > >> > > that
> > > > >> > > > > > >> > > > > the
> > > > >> > > > > > >> > > > > > internal changes would be much larger than
> I
> > > > >> expected.
> > > > >> > > > Some
> > > > >> > > > > > >> > protocols
> > > > >> > > > > > >> > > > > > involve clients, so they would require
> > changes
> > > > too.
> > > > >> > I'm
> > > > >> > > > > > thinking
> > > > >> > > > > > >> > that
> > > > >> > > > > > >> > > > for
> > > > >> > > > > > >> > > > > > now, removing the topic name from these
> > > requests
> > > > >> and
> > > > >> > > > > responses
> > > > >> > > > > > >> are
> > > > >> > > > > > >> > > out
> > > > >> > > > > > >> > > > of
> > > > >> > > > > > >> > > > > > scope.
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > > > I have decided to just keep the change
> > > > >> > > > LeaderAndIsrResponse
> > > > >> > > > > to
> > > > >> > > > > > >> > remove
> > > > >> > > > > > >> > > > the
> > > > >> > > > > > >> > > > > > topic name, and have updated the KIP to
> > reflect
> > > > >> this
> > > > >> > > > > change. I
> > > > >> > > > > > >> have
> > > > >> > > > > > >> > > > also
> > > > >> > > > > > >> > > > > > mentioned the other requests and responses
> in
> > > > >> future
> > > > >> > > work.
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > > > I'm hoping to start the voting process
> soon,
> > so
> > > > >> let me
> > > > >> > > > know
> > > > >> > > > > if
> > > > >> > > > > > >> > there
> > > > >> > > > > > >> > > is
> > > > >> > > > > > >> > > > > > anything else we should discuss.
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > > > Thank you,
> > > > >> > > > > > >> > > > > > Justine
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > > > On Tue, Sep 15, 2020 at 3:57 PM Justine
> > Olshan
> > > <
> > > > >> > > > > > >> > jols...@confluent.io
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > > > wrote:
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > > > > Hello again,
> > > > >> > > > > > >> > > > > > > To follow up on some of the other
> comments:
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > 10/11) We can remove the topic name from
> > > these
> > > > >> > > > > > >> > requests/responses,
> > > > >> > > > > > >> > > > and
> > > > >> > > > > > >> > > > > > > that means we will just have to make a
> few
> > > > >> internal
> > > > >> > > > > changes
> > > > >> > > > > > to
> > > > >> > > > > > >> > make
> > > > >> > > > > > >> > > > > > > partitions accessible by topic id and
> > > > partition.
> > > > >> I
> > > > >> > can
> > > > >> > > > > > update
> > > > >> > > > > > >> the
> > > > >> > > > > > >> > > KIP
> > > > >> > > > > > >> > > > > to
> > > > >> > > > > > >> > > > > > > remove them unless anyone thinks they
> > should
> > > > >> stay.
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > 12) Addressed in the previous email. I've
> > > > updated
> > > > >> > the
> > > > >> > > > KIP
> > > > >> > > > > to
> > > > >> > > > > > >> > > include
> > > > >> > > > > > >> > > > > > > tagged fields for the requests and
> > responses.
> > > > >> (More
> > > > >> > on
> > > > >> > > > > that
> > > > >> > > > > > >> > below)
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > 13) I think part of the idea for
> including
> > > this
> > > > >> > > > > information
> > > > >> > > > > > >> is to
> > > > >> > > > > > >> > > > > prepare
> > > > >> > > > > > >> > > > > > > for future changes. Perhaps the directory
> > > > >> structure
> > > > >> > > > might
> > > > >> > > > > > >> change
> > > > >> > > > > > >> > > from
> > > > >> > > > > > >> > > > > > > topicName_partitionNumber to something
> like
> > > > >> > > > > > >> > > topicID_partitionNumber.
> > > > >> > > > > > >> > > > > Then
> > > > >> > > > > > >> > > > > > > it would be useful to have the topic name
> > in
> > > > the
> > > > >> > file
> > > > >> > > > > since
> > > > >> > > > > > it
> > > > >> > > > > > >> > > would
> > > > >> > > > > > >> > > > > not
> > > > >> > > > > > >> > > > > > be
> > > > >> > > > > > >> > > > > > > in the directory structure. Supporting
> > topic
> > > > >> renames
> > > > >> > > > might
> > > > >> > > > > > be
> > > > >> > > > > > >> > > easier
> > > > >> > > > > > >> > > > if
> > > > >> > > > > > >> > > > > > the
> > > > >> > > > > > >> > > > > > > other fields are included. Would there be
> > any
> > > > >> > > downsides
> > > > >> > > > to
> > > > >> > > > > > >> > > including
> > > > >> > > > > > >> > > > > this
> > > > >> > > > > > >> > > > > > > information?
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > 14)  Yes, we would need to copy the
> > partition
> > > > >> > metadata
> > > > >> > > > > file
> > > > >> > > > > > in
> > > > >> > > > > > >> > this
> > > > >> > > > > > >> > > > > > > process. I've updated the KIP to include
> > > this.
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > 15) I believe Lucas meant v1 and v2 here.
> > He
> > > > was
> > > > >> > > > referring
> > > > >> > > > > > to
> > > > >> > > > > > >> how
> > > > >> > > > > > >> > > the
> > > > >> > > > > > >> > > > > > > requests would fall under different IBP
> and
> > > > meant
> > > > >> > that
> > > > >> > > > > older
> > > > >> > > > > > >> > > brokers
> > > > >> > > > > > >> > > > > > would
> > > > >> > > > > > >> > > > > > > have to use the older version of the
> > request
> > > > and
> > > > >> the
> > > > >> > > > > > existing
> > > > >> > > > > > >> > topic
> > > > >> > > > > > >> > > > > > > deletion process. At first, it seemed
> like
> > > > tagged
> > > > >> > > fields
> > > > >> > > > > > would
> > > > >> > > > > > >> > > > resolve
> > > > >> > > > > > >> > > > > > > the IBP issue. However, we may need IBP
> for
> > > > this
> > > > >> > > request
> > > > >> > > > > > after
> > > > >> > > > > > >> > all
> > > > >> > > > > > >> > > > > since
> > > > >> > > > > > >> > > > > > > the controller handles the topic deletion
> > > > >> > differently
> > > > >> > > > > > >> depending
> > > > >> > > > > > >> > on
> > > > >> > > > > > >> > > > the
> > > > >> > > > > > >> > > > > > IBP
> > > > >> > > > > > >> > > > > > > version. In an older version, we can't
> just
> > > > send
> > > > >> a
> > > > >> > > > > > StopReplica
> > > > >> > > > > > >> > > delete
> > > > >> > > > > > >> > > > > the
> > > > >> > > > > > >> > > > > > > topic immediately like we'd want to for
> > this
> > > > KIP.
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > This makes me wonder if we want tagged
> > fields
> > > > on
> > > > >> all
> > > > >> > > the
> > > > >> > > > > > >> requests
> > > > >> > > > > > >> > > > after
> > > > >> > > > > > >> > > > > > > all. Let me know your thoughts!
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > Justine
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > On Tue, Sep 15, 2020 at 1:03 PM Justine
> > > Olshan
> > > > <
> > > > >> > > > > > >> > > jols...@confluent.io
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > > > wrote:
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > >> Hi all,
> > > > >> > > > > > >> > > > > > >> Jun brought up a good point in his last
> > > email
> > > > >> about
> > > > >> > > > > tagged
> > > > >> > > > > > >> > fields,
> > > > >> > > > > > >> > > > and
> > > > >> > > > > > >> > > > > > >> I've updated the KIP to reflect that the
> > > > >> changes to
> > > > >> > > > > > requests
> > > > >> > > > > > >> and
> > > > >> > > > > > >> > > > > > responses
> > > > >> > > > > > >> > > > > > >> will be in the form of tagged fields to
> > > avoid
> > > > >> > > changing
> > > > >> > > > > IBP.
> > > > >> > > > > > >> > > > > > >>
> > > > >> > > > > > >> > > > > > >> Jun: I plan on sending a followup email
> to
> > > > >> address
> > > > >> > > some
> > > > >> > > > > of
> > > > >> > > > > > >> the
> > > > >> > > > > > >> > > other
> > > > >> > > > > > >> > > > > > >> points.
> > > > >> > > > > > >> > > > > > >>
> > > > >> > > > > > >> > > > > > >> Thanks,
> > > > >> > > > > > >> > > > > > >> Justine
> > > > >> > > > > > >> > > > > > >>
> > > > >> > > > > > >> > > > > > >> On Mon, Sep 14, 2020 at 4:25 PM Jun Rao
> <
> > > > >> > > > > j...@confluent.io>
> > > > >> > > > > > >> > wrote:
> > > > >> > > > > > >> > > > > > >>
> > > > >> > > > > > >> > > > > > >>> Hi, Justine,
> > > > >> > > > > > >> > > > > > >>>
> > > > >> > > > > > >> > > > > > >>> Thanks for the updated KIP. A few
> > comments
> > > > >> below.
> > > > >> > > > > > >> > > > > > >>>
> > > > >> > > > > > >> > > > > > >>> 10. LeaderAndIsr Response: Do we need
> the
> > > > topic
> > > > >> > > name?
> > > > >> > > > > > >> > > > > > >>>
> > > > >> > > > > > >> > > > > > >>> 11. For the changed request/response,
> > other
> > > > >> than
> > > > >> > > > > > >> LeaderAndIsr,
> > > > >> > > > > > >> > > > > > >>> UpdateMetadata, Metadata, do we need to
> > > > include
> > > > >> > the
> > > > >> > > > > topic
> > > > >> > > > > > >> name?
> > > > >> > > > > > >> > > > > > >>>
> > > > >> > > > > > >> > > > > > >>> 12. It seems that upgrades don't
> require
> > > IBP.
> > > > >> Does
> > > > >> > > > that
> > > > >> > > > > > mean
> > > > >> > > > > > >> > the
> > > > >> > > > > > >> > > > new
> > > > >> > > > > > >> > > > > > >>> fields
> > > > >> > > > > > >> > > > > > >>> in all the request/response are added
> as
> > > > tagged
> > > > >> > > fields
> > > > >> > > > > > >> without
> > > > >> > > > > > >> > > > > bumping
> > > > >> > > > > > >> > > > > > up
> > > > >> > > > > > >> > > > > > >>> the request version? It would be useful
> > to
> > > > make
> > > > >> > that
> > > > >> > > > > > clear.
> > > > >> > > > > > >> > > > > > >>>
> > > > >> > > > > > >> > > > > > >>> 13. Partition Metadata file: Do we need
> > to
> > > > >> include
> > > > >> > > the
> > > > >> > > > > > topic
> > > > >> > > > > > >> > name
> > > > >> > > > > > >> > > > and
> > > > >> > > > > > >> > > > > > the
> > > > >> > > > > > >> > > > > > >>> partition id since they are implied in
> > the
> > > > >> > directory
> > > > >> > > > > name?
> > > > >> > > > > > >> > > > > > >>>
> > > > >> > > > > > >> > > > > > >>> 14. In the JBOD mode, we support
> moving a
> > > > >> > > partition's
> > > > >> > > > > data
> > > > >> > > > > > >> from
> > > > >> > > > > > >> > > one
> > > > >> > > > > > >> > > > > > disk
> > > > >> > > > > > >> > > > > > >>> to
> > > > >> > > > > > >> > > > > > >>> another. Will the new partition
> metadata
> > > file
> > > > >> be
> > > > >> > > > copied
> > > > >> > > > > > >> during
> > > > >> > > > > > >> > > that
> > > > >> > > > > > >> > > > > > >>> process?
> > > > >> > > > > > >> > > > > > >>>
> > > > >> > > > > > >> > > > > > >>> 15. The KIP says "Remove deleted topics
> > > from
> > > > >> > > replicas
> > > > >> > > > by
> > > > >> > > > > > >> > sending
> > > > >> > > > > > >> > > > > > >>> StopReplicaRequest V2 for any topics
> > which
> > > do
> > > > >> not
> > > > >> > > > > contain
> > > > >> > > > > > a
> > > > >> > > > > > >> > topic
> > > > >> > > > > > >> > > > ID,
> > > > >> > > > > > >> > > > > > and
> > > > >> > > > > > >> > > > > > >>> V3 for any topics which do contain a
> > topic
> > > > >> ID.".
> > > > >> > > > > However,
> > > > >> > > > > > it
> > > > >> > > > > > >> > > seems
> > > > >> > > > > > >> > > > > the
> > > > >> > > > > > >> > > > > > >>> updated controller will create all
> > missing
> > > > >> topic
> > > > >> > IDs
> > > > >> > > > > first
> > > > >> > > > > > >> > before
> > > > >> > > > > > >> > > > > doing
> > > > >> > > > > > >> > > > > > >>> other actions. So, is
> StopReplicaRequest
> > V2
> > > > >> > needed?
> > > > >> > > > > > >> > > > > > >>>
> > > > >> > > > > > >> > > > > > >>> Jun
> > > > >> > > > > > >> > > > > > >>>
> > > > >> > > > > > >> > > > > > >>> On Fri, Sep 11, 2020 at 10:31 AM John
> > > > Roesler <
> > > > >> > > > > > >> > > vvcep...@apache.org
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > > >>> wrote:
> > > > >> > > > > > >> > > > > > >>>
> > > > >> > > > > > >> > > > > > >>> > Thanks, Justine!
> > > > >> > > > > > >> > > > > > >>> >
> > > > >> > > > > > >> > > > > > >>> > Your response seems compelling to me.
> > > > >> > > > > > >> > > > > > >>> >
> > > > >> > > > > > >> > > > > > >>> > -John
> > > > >> > > > > > >> > > > > > >>> >
> > > > >> > > > > > >> > > > > > >>> > On Fri, 2020-09-11 at 10:17 -0700,
> > > Justine
> > > > >> > Olshan
> > > > >> > > > > wrote:
> > > > >> > > > > > >> > > > > > >>> > > Hello all,
> > > > >> > > > > > >> > > > > > >>> > > Thanks for continuing the
> > discussion! I
> > > > >> have a
> > > > >> > > few
> > > > >> > > > > > >> > responses
> > > > >> > > > > > >> > > to
> > > > >> > > > > > >> > > > > > your
> > > > >> > > > > > >> > > > > > >>> > points.
> > > > >> > > > > > >> > > > > > >>> > >
> > > > >> > > > > > >> > > > > > >>> > > Tom: You are correct in that this
> KIP
> > > has
> > > > >> not
> > > > >> > > > > > mentioned
> > > > >> > > > > > >> the
> > > > >> > > > > > >> > > > > > >>> > > DeleteTopicsRequest. I think that
> > this
> > > > >> would
> > > > >> > be
> > > > >> > > > out
> > > > >> > > > > of
> > > > >> > > > > > >> > scope
> > > > >> > > > > > >> > > > for
> > > > >> > > > > > >> > > > > > >>> now, but
> > > > >> > > > > > >> > > > > > >>> > > may be something worth adding in
> the
> > > > >> future.
> > > > >> > > > > > >> > > > > > >>> > >
> > > > >> > > > > > >> > > > > > >>> > > John: We did consider sequence ids,
> > but
> > > > >> there
> > > > >> > > are
> > > > >> > > > a
> > > > >> > > > > > few
> > > > >> > > > > > >> > > reasons
> > > > >> > > > > > >> > > > > to
> > > > >> > > > > > >> > > > > > >>> favor
> > > > >> > > > > > >> > > > > > >>> > > UUIDs. There are several cases
> where
> > > > topics
> > > > >> > from
> > > > >> > > > > > >> different
> > > > >> > > > > > >> > > > > clusters
> > > > >> > > > > > >> > > > > > >>> may
> > > > >> > > > > > >> > > > > > >>> > > interact now and in the future. For
> > > > >> example,
> > > > >> > > > Mirror
> > > > >> > > > > > >> Maker 2
> > > > >> > > > > > >> > > may
> > > > >> > > > > > >> > > > > > >>> benefit
> > > > >> > > > > > >> > > > > > >>> > > from being able to detect when a
> > > cluster
> > > > >> being
> > > > >> > > > > > mirrored
> > > > >> > > > > > >> is
> > > > >> > > > > > >> > > > > deleted
> > > > >> > > > > > >> > > > > > >>> and
> > > > >> > > > > > >> > > > > > >>> > > recreated and globally unique
> > > identifiers
> > > > >> > would
> > > > >> > > > make
> > > > >> > > > > > >> > > resolving
> > > > >> > > > > > >> > > > > > issues
> > > > >> > > > > > >> > > > > > >>> > > easier than sequence IDs which may
> > > > collide
> > > > >> > > between
> > > > >> > > > > > >> > clusters.
> > > > >> > > > > > >> > > > > > KIP-405
> > > > >> > > > > > >> > > > > > >>> > > (tiered storage) will also benefit
> > from
> > > > >> > globally
> > > > >> > > > > > unique
> > > > >> > > > > > >> IDs
> > > > >> > > > > > >> > > as
> > > > >> > > > > > >> > > > > > shared
> > > > >> > > > > > >> > > > > > >>> > > buckets may be used between
> clusters.
> > > > >> > > > > > >> > > > > > >>> > >
> > > > >> > > > > > >> > > > > > >>> > > Globally unique IDs would also make
> > > > >> > > functionality
> > > > >> > > > > like
> > > > >> > > > > > >> > moving
> > > > >> > > > > > >> > > > > > topics
> > > > >> > > > > > >> > > > > > >>> > > between disparate clusters easier
> in
> > > the
> > > > >> > future,
> > > > >> > > > > > >> simplify
> > > > >> > > > > > >> > any
> > > > >> > > > > > >> > > > > > future
> > > > >> > > > > > >> > > > > > >>> > > implementations of backups and
> > > restores,
> > > > >> and
> > > > >> > > more.
> > > > >> > > > > In
> > > > >> > > > > > >> > > general,
> > > > >> > > > > > >> > > > > > >>> unique IDs
> > > > >> > > > > > >> > > > > > >>> > > would ensure that the source
> cluster
> > > > >> topics do
> > > > >> > > not
> > > > >> > > > > > >> conflict
> > > > >> > > > > > >> > > > with
> > > > >> > > > > > >> > > > > > the
> > > > >> > > > > > >> > > > > > >>> > > destination cluster topics.
> > > > >> > > > > > >> > > > > > >>> > >
> > > > >> > > > > > >> > > > > > >>> > > If we were to use sequence ids, we
> > > would
> > > > >> need
> > > > >> > > > > > >> sufficiently
> > > > >> > > > > > >> > > > large
> > > > >> > > > > > >> > > > > > >>> cluster
> > > > >> > > > > > >> > > > > > >>> > > ids to be stored with the topic
> > > > >> identifiers or
> > > > >> > > we
> > > > >> > > > > run
> > > > >> > > > > > >> the
> > > > >> > > > > > >> > > risk
> > > > >> > > > > > >> > > > of
> > > > >> > > > > > >> > > > > > >>> > > collisions. This will give up any
> > > > >> advantage in
> > > > >> > > > > > >> compactness
> > > > >> > > > > > >> > > that
> > > > >> > > > > > >> > > > > > >>> sequence
> > > > >> > > > > > >> > > > > > >>> > > numbers may bring. Given these
> > > > advantages I
> > > > >> > > think
> > > > >> > > > it
> > > > >> > > > > > >> makes
> > > > >> > > > > > >> > > > sense
> > > > >> > > > > > >> > > > > to
> > > > >> > > > > > >> > > > > > >>> use
> > > > >> > > > > > >> > > > > > >>> > > UUIDs.
> > > > >> > > > > > >> > > > > > >>> > >
> > > > >> > > > > > >> > > > > > >>> > > Gokul: This is an interesting idea,
> > but
> > > > >> this
> > > > >> > is
> > > > >> > > a
> > > > >> > > > > > >> breaking
> > > > >> > > > > > >> > > > > change.
> > > > >> > > > > > >> > > > > > >>> Out of
> > > > >> > > > > > >> > > > > > >>> > > scope for now, but maybe worth
> > > discussing
> > > > >> in
> > > > >> > the
> > > > >> > > > > > future.
> > > > >> > > > > > >> > > > > > >>> > >
> > > > >> > > > > > >> > > > > > >>> > > Hope this explains some of the
> > > decisions,
> > > > >> > > > > > >> > > > > > >>> > >
> > > > >> > > > > > >> > > > > > >>> > > Justine
> > > > >> > > > > > >> > > > > > >>> > >
> > > > >> > > > > > >> > > > > > >>> > >
> > > > >> > > > > > >> > > > > > >>> > >
> > > > >> > > > > > >> > > > > > >>> > > On Fri, Sep 11, 2020 at 8:27 AM
> Gokul
> > > > >> Ramanan
> > > > >> > > > > > >> Subramanian <
> > > > >> > > > > > >> > > > > > >>> > > gokul24...@gmail.com> wrote:
> > > > >> > > > > > >> > > > > > >>> > >
> > > > >> > > > > > >> > > > > > >>> > > > Hi.
> > > > >> > > > > > >> > > > > > >>> > > >
> > > > >> > > > > > >> > > > > > >>> > > > Thanks for the KIP.
> > > > >> > > > > > >> > > > > > >>> > > >
> > > > >> > > > > > >> > > > > > >>> > > > Have you thought about whether it
> > > makes
> > > > >> > sense
> > > > >> > > to
> > > > >> > > > > > >> support
> > > > >> > > > > > >> > > > > > >>> authorizing a
> > > > >> > > > > > >> > > > > > >>> > > > principal for a topic ID rather
> > than
> > > a
> > > > >> topic
> > > > >> > > > name
> > > > >> > > > > to
> > > > >> > > > > > >> > > achieve
> > > > >> > > > > > >> > > > > > >>> tighter
> > > > >> > > > > > >> > > > > > >>> > > > security?
> > > > >> > > > > > >> > > > > > >>> > > >
> > > > >> > > > > > >> > > > > > >>> > > > Or is the topic ID fundamentally
> an
> > > > >> internal
> > > > >> > > > > detail
> > > > >> > > > > > >> > similar
> > > > >> > > > > > >> > > > to
> > > > >> > > > > > >> > > > > > >>> epochs
> > > > >> > > > > > >> > > > > > >>> > used
> > > > >> > > > > > >> > > > > > >>> > > > in a bunch of other places in
> > Kafka?
> > > > >> > > > > > >> > > > > > >>> > > >
> > > > >> > > > > > >> > > > > > >>> > > > Thanks.
> > > > >> > > > > > >> > > > > > >>> > > >
> > > > >> > > > > > >> > > > > > >>> > > > On Fri, Sep 11, 2020 at 4:06 PM
> > John
> > > > >> > Roesler <
> > > > >> > > > > > >> > > > > > vvcep...@apache.org>
> > > > >> > > > > > >> > > > > > >>> > wrote:
> > > > >> > > > > > >> > > > > > >>> > > >
> > > > >> > > > > > >> > > > > > >>> > > > > Hello Justine,
> > > > >> > > > > > >> > > > > > >>> > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > Thanks for the KIP!
> > > > >> > > > > > >> > > > > > >>> > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > I happen to have been
> confronted
> > > > >> recently
> > > > >> > > with
> > > > >> > > > > the
> > > > >> > > > > > >> need
> > > > >> > > > > > >> > > to
> > > > >> > > > > > >> > > > > keep
> > > > >> > > > > > >> > > > > > >>> > track of
> > > > >> > > > > > >> > > > > > >>> > > > a
> > > > >> > > > > > >> > > > > > >>> > > > > large number of topics as
> > compactly
> > > > as
> > > > >> > > > > possible. I
> > > > >> > > > > > >> was
> > > > >> > > > > > >> > > > going
> > > > >> > > > > > >> > > > > to
> > > > >> > > > > > >> > > > > > >>> come
> > > > >> > > > > > >> > > > > > >>> > up
> > > > >> > > > > > >> > > > > > >>> > > > > with some way to dictionary
> > encode
> > > > the
> > > > >> > topic
> > > > >> > > > > names
> > > > >> > > > > > >> as
> > > > >> > > > > > >> > > > > integers,
> > > > >> > > > > > >> > > > > > >>> but
> > > > >> > > > > > >> > > > > > >>> > this
> > > > >> > > > > > >> > > > > > >>> > > > > seems much better!
> > > > >> > > > > > >> > > > > > >>> > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > Apologies if this has been
> raised
> > > > >> before,
> > > > >> > > but
> > > > >> > > > > I’m
> > > > >> > > > > > >> > > wondering
> > > > >> > > > > > >> > > > > > >>> about the
> > > > >> > > > > > >> > > > > > >>> > > > > choice of UUID vs sequence
> number
> > > for
> > > > >> the
> > > > >> > > ids.
> > > > >> > > > > > >> > Typically,
> > > > >> > > > > > >> > > > > I’ve
> > > > >> > > > > > >> > > > > > >>> seen
> > > > >> > > > > > >> > > > > > >>> > UUIDs
> > > > >> > > > > > >> > > > > > >>> > > > > in two situations:
> > > > >> > > > > > >> > > > > > >>> > > > > 1. When processes need to
> > generate
> > > > >> > > > non-colliding
> > > > >> > > > > > >> > > > identifiers
> > > > >> > > > > > >> > > > > > >>> without
> > > > >> > > > > > >> > > > > > >>> > > > > coordination.
> > > > >> > > > > > >> > > > > > >>> > > > > 2. When the identifier needs to
> > be
> > > > >> > > > “universally
> > > > >> > > > > > >> > unique”;
> > > > >> > > > > > >> > > > > I.e.,
> > > > >> > > > > > >> > > > > > >>> the
> > > > >> > > > > > >> > > > > > >>> > > > > identifier needs to distinguish
> > the
> > > > >> entity
> > > > >> > > > from
> > > > >> > > > > > all
> > > > >> > > > > > >> > other
> > > > >> > > > > > >> > > > > > >>> entities
> > > > >> > > > > > >> > > > > > >>> > that
> > > > >> > > > > > >> > > > > > >>> > > > > could ever exist. This is
> useful
> > in
> > > > >> cases
> > > > >> > > > where
> > > > >> > > > > > >> > entities
> > > > >> > > > > > >> > > > from
> > > > >> > > > > > >> > > > > > all
> > > > >> > > > > > >> > > > > > >>> > kinds
> > > > >> > > > > > >> > > > > > >>> > > > of
> > > > >> > > > > > >> > > > > > >>> > > > > systems get mixed together,
> such
> > as
> > > > >> when
> > > > >> > > > dumping
> > > > >> > > > > > >> logs
> > > > >> > > > > > >> > > from
> > > > >> > > > > > >> > > > > all
> > > > >> > > > > > >> > > > > > >>> > processes
> > > > >> > > > > > >> > > > > > >>> > > > in
> > > > >> > > > > > >> > > > > > >>> > > > > a company into a common system.
> > > > >> > > > > > >> > > > > > >>> > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > Maybe I’m being short-sighted,
> > but
> > > it
> > > > >> > > doesn’t
> > > > >> > > > > seem
> > > > >> > > > > > >> like
> > > > >> > > > > > >> > > > > either
> > > > >> > > > > > >> > > > > > >>> really
> > > > >> > > > > > >> > > > > > >>> > > > > applies here. It seems like the
> > > > brokers
> > > > >> > > could
> > > > >> > > > > and
> > > > >> > > > > > >> would
> > > > >> > > > > > >> > > > > achieve
> > > > >> > > > > > >> > > > > > >>> > consensus
> > > > >> > > > > > >> > > > > > >>> > > > > when creating a topic anyway,
> > which
> > > > is
> > > > >> all
> > > > >> > > > > that’s
> > > > >> > > > > > >> > > required
> > > > >> > > > > > >> > > > to
> > > > >> > > > > > >> > > > > > >>> > generate
> > > > >> > > > > > >> > > > > > >>> > > > > non-colliding sequence ids. For
> > the
> > > > >> > second,
> > > > >> > > as
> > > > >> > > > > you
> > > > >> > > > > > >> > > mention,
> > > > >> > > > > > >> > > > > we
> > > > >> > > > > > >> > > > > > >>> could
> > > > >> > > > > > >> > > > > > >>> > > > assign
> > > > >> > > > > > >> > > > > > >>> > > > > a UUID to the cluster as a
> whole,
> > > > which
> > > > >> > > would
> > > > >> > > > > > render
> > > > >> > > > > > >> > any
> > > > >> > > > > > >> > > > > > resource
> > > > >> > > > > > >> > > > > > >>> > scoped
> > > > >> > > > > > >> > > > > > >>> > > > to
> > > > >> > > > > > >> > > > > > >>> > > > > the broker universally unique
> as
> > > > well.
> > > > >> > > > > > >> > > > > > >>> > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > The reason I mention this is
> > that,
> > > > >> > although
> > > > >> > > a
> > > > >> > > > > UUID
> > > > >> > > > > > >> is
> > > > >> > > > > > >> > way
> > > > >> > > > > > >> > > > > more
> > > > >> > > > > > >> > > > > > >>> > compact
> > > > >> > > > > > >> > > > > > >>> > > > > than topic names, it’s still 16
> > > > bytes.
> > > > >> In
> > > > >> > > > > > contrast,
> > > > >> > > > > > >> a
> > > > >> > > > > > >> > > > 4-byte
> > > > >> > > > > > >> > > > > > >>> integer
> > > > >> > > > > > >> > > > > > >>> > > > > sequence id would give us 4
> > billion
> > > > >> unique
> > > > >> > > > > topics
> > > > >> > > > > > >> per
> > > > >> > > > > > >> > > > > cluster,
> > > > >> > > > > > >> > > > > > >>> which
> > > > >> > > > > > >> > > > > > >>> > > > seems
> > > > >> > > > > > >> > > > > > >>> > > > > like enough ;)
> > > > >> > > > > > >> > > > > > >>> > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > Considering the number of
> > different
> > > > >> times
> > > > >> > > > these
> > > > >> > > > > > >> topic
> > > > >> > > > > > >> > > > > > >>> identifiers are
> > > > >> > > > > > >> > > > > > >>> > > > sent
> > > > >> > > > > > >> > > > > > >>> > > > > over the wire or stored in
> > memory,
> > > it
> > > > >> > seems
> > > > >> > > > like
> > > > >> > > > > > it
> > > > >> > > > > > >> > might
> > > > >> > > > > > >> > > > be
> > > > >> > > > > > >> > > > > > >>> worth
> > > > >> > > > > > >> > > > > > >>> > the
> > > > >> > > > > > >> > > > > > >>> > > > > additional 4x space savings.
> > > > >> > > > > > >> > > > > > >>> > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > What do you think about this?
> > > > >> > > > > > >> > > > > > >>> > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > Thanks,
> > > > >> > > > > > >> > > > > > >>> > > > > John
> > > > >> > > > > > >> > > > > > >>> > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > On Fri, Sep 11, 2020, at 03:20,
> > Tom
> > > > >> > Bentley
> > > > >> > > > > wrote:
> > > > >> > > > > > >> > > > > > >>> > > > > > Hi Justine,
> > > > >> > > > > > >> > > > > > >>> > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > This looks like a very
> welcome
> > > > >> > > improvement.
> > > > >> > > > > > >> Thanks!
> > > > >> > > > > > >> > > > > > >>> > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > Maybe I missed it, but the
> KIP
> > > > >> doesn't
> > > > >> > > seem
> > > > >> > > > to
> > > > >> > > > > > >> > mention
> > > > >> > > > > > >> > > > > > changing
> > > > >> > > > > > >> > > > > > >>> > > > > > DeleteTopicsRequest to
> identify
> > > the
> > > > >> > topic
> > > > >> > > > > using
> > > > >> > > > > > an
> > > > >> > > > > > >> > id.
> > > > >> > > > > > >> > > > > Maybe
> > > > >> > > > > > >> > > > > > >>> > that's out
> > > > >> > > > > > >> > > > > > >>> > > > > of
> > > > >> > > > > > >> > > > > > >>> > > > > > scope, but
> DeleteTopicsRequest
> > is
> > > > not
> > > > >> > > listed
> > > > >> > > > > > among
> > > > >> > > > > > >> > the
> > > > >> > > > > > >> > > > > Future
> > > > >> > > > > > >> > > > > > >>> Work
> > > > >> > > > > > >> > > > > > >>> > APIs
> > > > >> > > > > > >> > > > > > >>> > > > > > either.
> > > > >> > > > > > >> > > > > > >>> > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > Kind regards,
> > > > >> > > > > > >> > > > > > >>> > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > Tom
> > > > >> > > > > > >> > > > > > >>> > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > On Thu, Sep 10, 2020 at 3:59
> PM
> > > > >> Satish
> > > > >> > > > > Duggana <
> > > > >> > > > > > >> > > > > > >>> > > > satish.dugg...@gmail.com
> > > > >> > > > > > >> > > > > > >>> > > > > > wrote:
> > > > >> > > > > > >> > > > > > >>> > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > Thanks Lucas/Justine for
> the
> > > nice
> > > > >> KIP.
> > > > >> > > > > > >> > > > > > >>> > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > It has several benefits
> which
> > > > also
> > > > >> > > include
> > > > >> > > > > > >> > > simplifying
> > > > >> > > > > > >> > > > > the
> > > > >> > > > > > >> > > > > > >>> topic
> > > > >> > > > > > >> > > > > > >>> > > > > > > deletion process by
> > controller
> > > > and
> > > > >> > logs
> > > > >> > > > > > cleanup
> > > > >> > > > > > >> by
> > > > >> > > > > > >> > > > > brokers
> > > > >> > > > > > >> > > > > > in
> > > > >> > > > > > >> > > > > > >>> > corner
> > > > >> > > > > > >> > > > > > >>> > > > > > > cases.
> > > > >> > > > > > >> > > > > > >>> > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > Best,
> > > > >> > > > > > >> > > > > > >>> > > > > > > Satish.
> > > > >> > > > > > >> > > > > > >>> > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > On Wed, Sep 9, 2020 at
> 10:07
> > PM
> > > > >> > Justine
> > > > >> > > > > > Olshan <
> > > > >> > > > > > >> > > > > > >>> > jols...@confluent.io
> > > > >> > > > > > >> > > > > > >>> > > > > > > wrote:
> > > > >> > > > > > >> > > > > > >>> > > > > > > > Hello all, it's been
> > almost a
> > > > >> year!
> > > > >> > > I've
> > > > >> > > > > > made
> > > > >> > > > > > >> > some
> > > > >> > > > > > >> > > > > > changes
> > > > >> > > > > > >> > > > > > >>> to
> > > > >> > > > > > >> > > > > > >>> > this
> > > > >> > > > > > >> > > > > > >>> > > > > KIP
> > > > >> > > > > > >> > > > > > >>> > > > > > > and hope to continue the
> > > > >> discussion.
> > > > >> > > > > > >> > > > > > >>> > > > > > > > One of the main changes
> > I've
> > > > >> added
> > > > >> > is
> > > > >> > > > now
> > > > >> > > > > > the
> > > > >> > > > > > >> > > > metadata
> > > > >> > > > > > >> > > > > > >>> response
> > > > >> > > > > > >> > > > > > >>> > > > will
> > > > >> > > > > > >> > > > > > >>> > > > > > > include the topic ID (as
> > Colin
> > > > >> > > suggested).
> > > > >> > > > > > >> Clients
> > > > >> > > > > > >> > > can
> > > > >> > > > > > >> > > > > > >>> obtain the
> > > > >> > > > > > >> > > > > > >>> > > > > topicID
> > > > >> > > > > > >> > > > > > >>> > > > > > > of a given topic through a
> > > > >> > > > TopicDescription.
> > > > >> > > > > > The
> > > > >> > > > > > >> > > > topicId
> > > > >> > > > > > >> > > > > > will
> > > > >> > > > > > >> > > > > > >>> > also be
> > > > >> > > > > > >> > > > > > >>> > > > > > > included with the
> > > UpdateMetadata
> > > > >> > > request.
> > > > >> > > > > > >> > > > > > >>> > > > > > > > Let me know what you all
> > > think.
> > > > >> > > > > > >> > > > > > >>> > > > > > > > Thank you,
> > > > >> > > > > > >> > > > > > >>> > > > > > > > Justine
> > > > >> > > > > > >> > > > > > >>> > > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > > On 2019/09/13 16:38:26,
> > > "Colin
> > > > >> > > McCabe" <
> > > > >> > > > > > >> > > > > > cmcc...@apache.org
> > > > >> > > > > > >> > > > > > >>> >
> > > > >> > > > > > >> > > > > > >>> > wrote:
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > Hi Lucas,
> > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > Thanks for tackling
> this.
> > > > >> Topic
> > > > >> > IDs
> > > > >> > > > > are a
> > > > >> > > > > > >> > great
> > > > >> > > > > > >> > > > > idea,
> > > > >> > > > > > >> > > > > > >>> and
> > > > >> > > > > > >> > > > > > >>> > this
> > > > >> > > > > > >> > > > > > >>> > > > is
> > > > >> > > > > > >> > > > > > >>> > > > > a
> > > > >> > > > > > >> > > > > > >>> > > > > > > really good writeup.
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > For
> > > /brokers/topics/[topic],
> > > > >> the
> > > > >> > > > schema
> > > > >> > > > > > >> version
> > > > >> > > > > > >> > > > > should
> > > > >> > > > > > >> > > > > > be
> > > > >> > > > > > >> > > > > > >>> > bumped
> > > > >> > > > > > >> > > > > > >>> > > > to
> > > > >> > > > > > >> > > > > > >>> > > > > > > version 3, rather than 2.
> > > > KIP-455
> > > > >> > > bumped
> > > > >> > > > > the
> > > > >> > > > > > >> > version
> > > > >> > > > > > >> > > > of
> > > > >> > > > > > >> > > > > > this
> > > > >> > > > > > >> > > > > > >>> > znode
> > > > >> > > > > > >> > > > > > >>> > > > to
> > > > >> > > > > > >> > > > > > >>> > > > > 2
> > > > >> > > > > > >> > > > > > >>> > > > > > > already :)
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > Given that we're going
> to
> > > be
> > > > >> > seeing
> > > > >> > > > > these
> > > > >> > > > > > >> > things
> > > > >> > > > > > >> > > as
> > > > >> > > > > > >> > > > > > >>> strings
> > > > >> > > > > > >> > > > > > >>> > as
> > > > >> > > > > > >> > > > > > >>> > > > lot
> > > > >> > > > > > >> > > > > > >>> > > > > (in
> > > > >> > > > > > >> > > > > > >>> > > > > > > logs, in ZooKeeper, on the
> > > > >> > command-line,
> > > > >> > > > > > etc.),
> > > > >> > > > > > >> > does
> > > > >> > > > > > >> > > it
> > > > >> > > > > > >> > > > > > make
> > > > >> > > > > > >> > > > > > >>> > sense to
> > > > >> > > > > > >> > > > > > >>> > > > > use
> > > > >> > > > > > >> > > > > > >>> > > > > > > base64 when converting them
> > to
> > > > >> > strings?
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > Here is an example of
> the
> > > hex
> > > > >> > > > > > >> representation:
> > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > > >> > 6fcb514b-b878-4c9d-95b7-8dc3a7ce6fd8
> > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > And here is an example
> in
> > > > >> base64.
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > b8tRS7h4TJ2Vt43Dp85v2A
> > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > The base64 version
> saves
> > 15
> > > > >> > letters
> > > > >> > > > (to
> > > > >> > > > > be
> > > > >> > > > > > >> > fair,
> > > > >> > > > > > >> > > 4
> > > > >> > > > > > >> > > > of
> > > > >> > > > > > >> > > > > > >>> those
> > > > >> > > > > > >> > > > > > >>> > were
> > > > >> > > > > > >> > > > > > >>> > > > > > > dashes that we could have
> > > elided
> > > > in
> > > > >> > the
> > > > >> > > > hex
> > > > >> > > > > > >> > > > > > representation.)
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > Another thing to
> consider
> > > is
> > > > >> that
> > > > >> > we
> > > > >> > > > > > should
> > > > >> > > > > > >> > > specify
> > > > >> > > > > > >> > > > > > that
> > > > >> > > > > > >> > > > > > >>> the
> > > > >> > > > > > >> > > > > > >>> > > > > > > all-zeroes UUID is not a
> > valid
> > > > >> topic
> > > > >> > > UUID.
> > > > >> > > > > >  We
> > > > >> > > > > > >> > can't
> > > > >> > > > > > >> > > > use
> > > > >> > > > > > >> > > > > > >>> null
> > > > >> > > > > > >> > > > > > >>> > for
> > > > >> > > > > > >> > > > > > >>> > > > this
> > > > >> > > > > > >> > > > > > >>> > > > > > > because we can't pass a
> null
> > > UUID
> > > > >> over
> > > > >> > > the
> > > > >> > > > > RPC
> > > > >> > > > > > >> > > protocol
> > > > >> > > > > > >> > > > > > >>> (there
> > > > >> > > > > > >> > > > > > >>> > is no
> > > > >> > > > > > >> > > > > > >>> > > > > > > special pattern for null,
> nor
> > > do
> > > > we
> > > > >> > want
> > > > >> > > > to
> > > > >> > > > > > >> waste
> > > > >> > > > > > >> > > space
> > > > >> > > > > > >> > > > > > >>> reserving
> > > > >> > > > > > >> > > > > > >>> > > > such
> > > > >> > > > > > >> > > > > > >>> > > > > a
> > > > >> > > > > > >> > > > > > >>> > > > > > > pattern.)
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > Maybe I missed it, but
> > did
> > > > you
> > > > >> > > > describe
> > > > >> > > > > > >> > > "migration
> > > > >> > > > > > >> > > > > > of...
> > > > >> > > > > > >> > > > > > >>> > existing
> > > > >> > > > > > >> > > > > > >>> > > > > > > topic[s] without topic IDs"
> > in
> > > > >> detail
> > > > >> > in
> > > > >> > > > any
> > > > >> > > > > > >> > section?
> > > > >> > > > > > >> > > > It
> > > > >> > > > > > >> > > > > > >>> seems
> > > > >> > > > > > >> > > > > > >>> > like
> > > > >> > > > > > >> > > > > > >>> > > > > when
> > > > >> > > > > > >> > > > > > >>> > > > > > > the new controller becomes
> > > > active,
> > > > >> it
> > > > >> > > > should
> > > > >> > > > > > >> just
> > > > >> > > > > > >> > > > > generate
> > > > >> > > > > > >> > > > > > >>> random
> > > > >> > > > > > >> > > > > > >>> > > > > UUIDs for
> > > > >> > > > > > >> > > > > > >>> > > > > > > these, and write the random
> > > UUIDs
> > > > >> back
> > > > >> > > to
> > > > >> > > > > > >> > ZooKeeper.
> > > > >> > > > > > >> > > > It
> > > > >> > > > > > >> > > > > > >>> would be
> > > > >> > > > > > >> > > > > > >>> > > > good
> > > > >> > > > > > >> > > > > > >>> > > > > to
> > > > >> > > > > > >> > > > > > >>> > > > > > > spell that out.  We should
> > make
> > > > it
> > > > >> > clear
> > > > >> > > > > that
> > > > >> > > > > > >> this
> > > > >> > > > > > >> > > > > happens
> > > > >> > > > > > >> > > > > > >>> > regardless
> > > > >> > > > > > >> > > > > > >>> > > > > of
> > > > >> > > > > > >> > > > > > >>> > > > > > > the inter-broker protocol
> > > version
> > > > >> > (it's
> > > > >> > > a
> > > > >> > > > > > >> > compatible
> > > > >> > > > > > >> > > > > > change).
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > "LeaderAndIsrRequests
> > > > >> including an
> > > > >> > > > > > >> > > > is_every_partition
> > > > >> > > > > > >> > > > > > >>> flag"
> > > > >> > > > > > >> > > > > > >>> > > > seems a
> > > > >> > > > > > >> > > > > > >>> > > > > > > bit wordy.  Can we just
> call
> > > > these
> > > > >> > "full
> > > > >> > > > > > >> > > > > > >>> LeaderAndIsrRequests"?
> > > > >> > > > > > >> > > > > > >>> > Then
> > > > >> > > > > > >> > > > > > >>> > > > > the
> > > > >> > > > > > >> > > > > > >>> > > > > > > RPC field could be named
> > > "full".
> > > > >> > Also,
> > > > >> > > it
> > > > >> > > > > > would
> > > > >> > > > > > >> > > > probably
> > > > >> > > > > > >> > > > > > be
> > > > >> > > > > > >> > > > > > >>> > better
> > > > >> > > > > > >> > > > > > >>> > > > > for the
> > > > >> > > > > > >> > > > > > >>> > > > > > > RPC field to be an enum of
> {
> > > > >> > > UNSPECIFIED,
> > > > >> > > > > > >> > > INCREMENTAL,
> > > > >> > > > > > >> > > > > FULL
> > > > >> > > > > > >> > > > > > >>> }, so
> > > > >> > > > > > >> > > > > > >>> > > > that
> > > > >> > > > > > >> > > > > > >>> > > > > we
> > > > >> > > > > > >> > > > > > >>> > > > > > > can cleanly handle old
> > versions
> > > > (by
> > > > >> > > > treating
> > > > >> > > > > > >> them
> > > > >> > > > > > >> > as
> > > > >> > > > > > >> > > > > > >>> UNSPECIFIED)
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > In the
> > LeaderAndIsrRequest
> > > > >> > section,
> > > > >> > > > you
> > > > >> > > > > > >> write
> > > > >> > > > > > >> > "A
> > > > >> > > > > > >> > > > > final
> > > > >> > > > > > >> > > > > > >>> > deletion
> > > > >> > > > > > >> > > > > > >>> > > > > event
> > > > >> > > > > > >> > > > > > >>> > > > > > > will be secheduled for X ms
> > > after
> > > > >> the
> > > > >> > > > > > >> > > > LeaderAndIsrRequest
> > > > >> > > > > > >> > > > > > was
> > > > >> > > > > > >> > > > > > >>> > first
> > > > >> > > > > > >> > > > > > >>> > > > > > > received..."  I guess the X
> > > was a
> > > > >> > > > > placeholder
> > > > >> > > > > > >> that
> > > > >> > > > > > >> > > you
> > > > >> > > > > > >> > > > > > >>> intended
> > > > >> > > > > > >> > > > > > >>> > to
> > > > >> > > > > > >> > > > > > >>> > > > > replace
> > > > >> > > > > > >> > > > > > >>> > > > > > > before posting? :)  In any
> > > case,
> > > > >> this
> > > > >> > > > seems
> > > > >> > > > > > like
> > > > >> > > > > > >> > the
> > > > >> > > > > > >> > > > kind
> > > > >> > > > > > >> > > > > > of
> > > > >> > > > > > >> > > > > > >>> > thing
> > > > >> > > > > > >> > > > > > >>> > > > we'd
> > > > >> > > > > > >> > > > > > >>> > > > > > > want a configuration for.
> > > Let's
> > > > >> > > describe
> > > > >> > > > > that
> > > > >> > > > > > >> > > > > > configuration
> > > > >> > > > > > >> > > > > > >>> key
> > > > >> > > > > > >> > > > > > >>> > > > > somewhere
> > > > >> > > > > > >> > > > > > >>> > > > > > > in this KIP, including what
> > its
> > > > >> > default
> > > > >> > > > > value
> > > > >> > > > > > >> is.
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > We should probably also
> > > log a
> > > > >> > bunch
> > > > >> > > of
> > > > >> > > > > > >> messages
> > > > >> > > > > > >> > > at
> > > > >> > > > > > >> > > > > WARN
> > > > >> > > > > > >> > > > > > >>> level
> > > > >> > > > > > >> > > > > > >>> > > > when
> > > > >> > > > > > >> > > > > > >>> > > > > > > something is scheduled for
> > > > >> deletion,
> > > > >> > as
> > > > >> > > > > well.
> > > > >> > > > > > >> > (Maybe
> > > > >> > > > > > >> > > > > this
> > > > >> > > > > > >> > > > > > >>> was
> > > > >> > > > > > >> > > > > > >>> > > > > assumed, but
> > > > >> > > > > > >> > > > > > >>> > > > > > > it would be good to mention
> > > it).
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > I feel like there are a
> > few
> > > > >> > sections
> > > > >> > > > > that
> > > > >> > > > > > >> > should
> > > > >> > > > > > >> > > be
> > > > >> > > > > > >> > > > > > >>> moved to
> > > > >> > > > > > >> > > > > > >>> > > > > "rejected
> > > > >> > > > > > >> > > > > > >>> > > > > > > alternatives."  For
> example,
> > in
> > > > the
> > > > >> > > > > > DeleteTopics
> > > > >> > > > > > >> > > > section,
> > > > >> > > > > > >> > > > > > >>> since
> > > > >> > > > > > >> > > > > > >>> > we're
> > > > >> > > > > > >> > > > > > >>> > > > > not
> > > > >> > > > > > >> > > > > > >>> > > > > > > going to do option 1 or 2,
> > > these
> > > > >> > should
> > > > >> > > be
> > > > >> > > > > > moved
> > > > >> > > > > > >> > into
> > > > >> > > > > > >> > > > > > >>> "rejected
> > > > >> > > > > > >> > > > > > >>> > > > > > > alternatives,"  rather than
> > > > >> appearing
> > > > >> > > > > inline.
> > > > >> > > > > > >> > > Another
> > > > >> > > > > > >> > > > > case
> > > > >> > > > > > >> > > > > > >>> is
> > > > >> > > > > > >> > > > > > >>> > the
> > > > >> > > > > > >> > > > > > >>> > > > > "Should
> > > > >> > > > > > >> > > > > > >>> > > > > > > we remove topic name from
> the
> > > > >> protocol
> > > > >> > > > where
> > > > >> > > > > > >> > > possible"
> > > > >> > > > > > >> > > > > > >>> section.
> > > > >> > > > > > >> > > > > > >>> > This
> > > > >> > > > > > >> > > > > > >>> > > > > is
> > > > >> > > > > > >> > > > > > >>> > > > > > > clearly discussing a design
> > > > >> > alternative
> > > > >> > > > that
> > > > >> > > > > > >> we're
> > > > >> > > > > > >> > > not
> > > > >> > > > > > >> > > > > > >>> proposing
> > > > >> > > > > > >> > > > > > >>> > to
> > > > >> > > > > > >> > > > > > >>> > > > > > > implement: removing the
> topic
> > > > name
> > > > >> > from
> > > > >> > > > > those
> > > > >> > > > > > >> > > > protocols.
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > Is it really necessary
> to
> > > > have
> > > > >> a
> > > > >> > new
> > > > >> > > > > > >> > > > > > >>> > /admin/delete_topics_by_id
> > > > >> > > > > > >> > > > > > >>> > > > > path
> > > > >> > > > > > >> > > > > > >>> > > > > > > in ZooKeeper?  It seems
> like
> > we
> > > > >> don't
> > > > >> > > > really
> > > > >> > > > > > >> need
> > > > >> > > > > > >> > > this.
> > > > >> > > > > > >> > > > > > >>> Whenever
> > > > >> > > > > > >> > > > > > >>> > > > > there is
> > > > >> > > > > > >> > > > > > >>> > > > > > > a new controller, we'll
> send
> > > out
> > > > >> full
> > > > >> > > > > > >> > > > > LeaderAndIsrRequests
> > > > >> > > > > > >> > > > > > >>> which
> > > > >> > > > > > >> > > > > > >>> > will
> > > > >> > > > > > >> > > > > > >>> > > > > > > trigger the stale topics to
> > be
> > > > >> cleaned
> > > > >> > > up.
> > > > >> > > > > >  The
> > > > >> > > > > > >> > > active
> > > > >> > > > > > >> > > > > > >>> > controller
> > > > >> > > > > > >> > > > > > >>> > > > will
> > > > >> > > > > > >> > > > > > >>> > > > > > > also send the full
> > > > >> LeaderAndIsrRequest
> > > > >> > > to
> > > > >> > > > > > >> brokers
> > > > >> > > > > > >> > > that
> > > > >> > > > > > >> > > > > are
> > > > >> > > > > > >> > > > > > >>> just
> > > > >> > > > > > >> > > > > > >>> > > > > starting
> > > > >> > > > > > >> > > > > > >>> > > > > > > up.    So we don't really
> > need
> > > > this
> > > > >> > kind
> > > > >> > > > of
> > > > >> > > > > > >> > two-phase
> > > > >> > > > > > >> > > > > > commit
> > > > >> > > > > > >> > > > > > >>> > (send
> > > > >> > > > > > >> > > > > > >>> > > > out
> > > > >> > > > > > >> > > > > > >>> > > > > > > StopReplicasRequest, get
> ACKs
> > > > from
> > > > >> all
> > > > >> > > > > nodes,
> > > > >> > > > > > >> > commit
> > > > >> > > > > > >> > > by
> > > > >> > > > > > >> > > > > > >>> removing
> > > > >> > > > > > >> > > > > > >>> > > > > > > /admin/delete_topics node)
> > any
> > > > >> more.
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > You mention that
> > > FetchRequest
> > > > >> will
> > > > >> > > now
> > > > >> > > > > > >> include
> > > > >> > > > > > >> > > UUID
> > > > >> > > > > > >> > > > > to
> > > > >> > > > > > >> > > > > > >>> avoid
> > > > >> > > > > > >> > > > > > >>> > > > issues
> > > > >> > > > > > >> > > > > > >>> > > > > > > where requests are made to
> > > stale
> > > > >> > > > partitions.
> > > > >> > > > > > >> > > However,
> > > > >> > > > > > >> > > > > > >>> adding a
> > > > >> > > > > > >> > > > > > >>> > UUID
> > > > >> > > > > > >> > > > > > >>> > > > to
> > > > >> > > > > > >> > > > > > >>> > > > > > > MetadataRequest is listed
> as
> > > > future
> > > > >> > > work,
> > > > >> > > > > out
> > > > >> > > > > > of
> > > > >> > > > > > >> > > scope
> > > > >> > > > > > >> > > > > for
> > > > >> > > > > > >> > > > > > >>> this
> > > > >> > > > > > >> > > > > > >>> > KIP.
> > > > >> > > > > > >> > > > > > >>> > > > > How
> > > > >> > > > > > >> > > > > > >>> > > > > > > will the client learn what
> > the
> > > > >> topic
> > > > >> > > UUID
> > > > >> > > > > is,
> > > > >> > > > > > if
> > > > >> > > > > > >> > the
> > > > >> > > > > > >> > > > > > metadata
> > > > >> > > > > > >> > > > > > >>> > > > response
> > > > >> > > > > > >> > > > > > >>> > > > > > > doesn't include that
> > > information?
> > > > >> It
> > > > >> > > > seems
> > > > >> > > > > > like
> > > > >> > > > > > >> > > adding
> > > > >> > > > > > >> > > > > the
> > > > >> > > > > > >> > > > > > >>> UUID
> > > > >> > > > > > >> > > > > > >>> > to
> > > > >> > > > > > >> > > > > > >>> > > > > > > MetadataResponse would be
> an
> > > > >> > improvement
> > > > >> > > > > here
> > > > >> > > > > > >> that
> > > > >> > > > > > >> > > > might
> > > > >> > > > > > >> > > > > > not
> > > > >> > > > > > >> > > > > > >>> be
> > > > >> > > > > > >> > > > > > >>> > too
> > > > >> > > > > > >> > > > > > >>> > > > > hard to
> > > > >> > > > > > >> > > > > > >>> > > > > > > make.
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > best,
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > Colin
> > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > On Mon, Sep 9, 2019, at
> > > > 17:48,
> > > > >> > > Ryanne
> > > > >> > > > > > Dolan
> > > > >> > > > > > >> > > wrote:
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > Lucas, this would be
> > > great.
> > > > >> I've
> > > > >> > > run
> > > > >> > > > > > into
> > > > >> > > > > > >> > > issues
> > > > >> > > > > > >> > > > > with
> > > > >> > > > > > >> > > > > > >>> > topics
> > > > >> > > > > > >> > > > > > >>> > > > > being
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > resurrected
> > accidentally,
> > > > >> since
> > > > >> > a
> > > > >> > > > > client
> > > > >> > > > > > >> > cannot
> > > > >> > > > > > >> > > > > > easily
> > > > >> > > > > > >> > > > > > >>> > > > > distinguish
> > > > >> > > > > > >> > > > > > >>> > > > > > > between
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > a deleted topic and a
> > new
> > > > >> topic
> > > > >> > > with
> > > > >> > > > > the
> > > > >> > > > > > >> same
> > > > >> > > > > > >> > > > name.
> > > > >> > > > > > >> > > > > > I'd
> > > > >> > > > > > >> > > > > > >>> > need
> > > > >> > > > > > >> > > > > > >>> > > > the
> > > > >> > > > > > >> > > > > > >>> > > > > ID
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > accessible from the
> > > client
> > > > to
> > > > >> > > solve
> > > > >> > > > > that
> > > > >> > > > > > >> > issue,
> > > > >> > > > > > >> > > > but
> > > > >> > > > > > >> > > > > > >>> this
> > > > >> > > > > > >> > > > > > >>> > is a
> > > > >> > > > > > >> > > > > > >>> > > > > good
> > > > >> > > > > > >> > > > > > >>> > > > > > > first
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > step.
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > Ryanne
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > On Wed, Sep 4, 2019
> at
> > > 1:41
> > > > >> PM
> > > > >> > > Lucas
> > > > >> > > > > > >> > > Bradstreet <
> > > > >> > > > > > >> > > > > > >>> > > > > lu...@confluent.io>
> > > > >> > > > > > >> > > > > > >>> > > > > > > wrote:
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Hi all,
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > I would like to
> kick
> > > off
> > > > >> > > > discussion
> > > > >> > > > > of
> > > > >> > > > > > >> > > KIP-516,
> > > > >> > > > > > >> > > > > an
> > > > >> > > > > > >> > > > > > >>> > > > > implementation
> > > > >> > > > > > >> > > > > > >>> > > > > > > of topic
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > IDs for Kafka.
> Topic
> > > IDs
> > > > >> aim
> > > > >> > to
> > > > >> > > > > solve
> > > > >> > > > > > >> topic
> > > > >> > > > > > >> > > > > > >>> uniqueness
> > > > >> > > > > > >> > > > > > >>> > > > > problems in
> > > > >> > > > > > >> > > > > > >>> > > > > > > Kafka,
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > where referring to
> a
> > > > topic
> > > > >> by
> > > > >> > > name
> > > > >> > > > > > >> alone is
> > > > >> > > > > > >> > > > > > >>> insufficient.
> > > > >> > > > > > >> > > > > > >>> > > > Such
> > > > >> > > > > > >> > > > > > >>> > > > > > > cases
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > include when a
> topic
> > > has
> > > > >> been
> > > > >> > > > > deleted
> > > > >> > > > > > >> and
> > > > >> > > > > > >> > > > > recreated
> > > > >> > > > > > >> > > > > > >>> with
> > > > >> > > > > > >> > > > > > >>> > the
> > > > >> > > > > > >> > > > > > >>> > > > > same
> > > > >> > > > > > >> > > > > > >>> > > > > > > name.
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Unique identifiers
> > will
> > > > >> help
> > > > >> > > > > simplify
> > > > >> > > > > > >> and
> > > > >> > > > > > >> > > > improve
> > > > >> > > > > > >> > > > > > >>> Kafka's
> > > > >> > > > > > >> > > > > > >>> > > > topic
> > > > >> > > > > > >> > > > > > >>> > > > > > > deletion
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > process, as well as
> > > > prevent
> > > > >> > > cases
> > > > >> > > > > > where
> > > > >> > > > > > >> > > brokers
> > > > >> > > > > > >> > > > > may
> > > > >> > > > > > >> > > > > > >>> > > > incorrectly
> > > > >> > > > > > >> > > > > > >>> > > > > > > interact
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > with stale versions
> > of
> > > > >> topics.
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > >
> > > > >> > > > > > >> > > > > > >>> >
> > > > >> > > > > > >> > > > > > >>>
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> >
> > > > >> > > > > > >>
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Looking forward to
> > your
> > > > >> > > thoughts.
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Lucas
> > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > >>> >
> > > > >> > > > > > >> > > > > > >>> >
> > > > >> > > > > > >> > > > > > >>>
> > > > >> > > > > > >> > > > > > >>
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> >
> > > > >> > > > > > >>
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Reply via email to