Hi Justine,
Thanks for picking up this work. I have a few questions/comments:
1. It sounds like the directory structure is still going to be based on
topic names. Do I have that right? One complication is that the
LeaderAndIsr request does not include the topic name any longer. This means
that a replica must wait for the UpdateMetadata request to arrive with the
topic name mapping before it can create the directory. I wonder if it would
be simpler to avoid assumptions on the ordering of UpdateMetadata and let
LeaderAndIsr include the topic name as well. Feels like we are not saving
that much by excluding it.
2. On a related note, it seems that the reason we have the partition
metadata file is because we are not changing the directory structure. We
need it so that we remember which directories map to which topic id. I
think it would be useful to add some detail about why changing the
directory layout was rejected. Basically trying to understand how much
effort we are saving by skipping this step if we have to implement this
additional bookkeeping. One concern I have for example is that the
partition metadata file gets corrupt and then all bets are off as far as
topic consistency.
3. Is it worth adding support now for topic ids in the `Produce` request
now? Overhead is mentioned as one of the motivations and the best APIs to
get savings from are `Produce` and `Fetch`.
4. When it comes to bootstrapping the metadata topic with respect to
KIP-500, one suggestion would be to reserve a sentinel topic ID which can
be used initially by new brokers. When the first controller is elected, it
can assign a topicId to the metadata topic. This is a bit similar to how we
were planning to generate the clusterId.
Thanks,
Jason
On Thu, Sep 24, 2020 at 11:10 AM Jun Rao <j...@confluent.io> wrote:
> Hi, Justine,
>
> Thanks for the updated KIP. A few more comments below.
>
> 30. {"name": "id", "type": "string", "doc": "version id"}}: The doc should
> say UUID.
>
> 31. LeaderAndIsrResponse v5 and StopReplicaResponse v4 : It seems there is
> no need to add topic_id at partitions level.
>
> 32. Regarding partition metadata file. Perhaps the key can be a single
> word, sth like the following.
> version: 0
> topic_id: 46bdb63f-9e8d-4a38-bf7b-ee4eb2a794e4
>
> 33. Another tricky thing that I realized is how to support the metadata
> topic introduced in KIP-595. It's a bit tricky to assign a UUID to the
> metadata topic since we have a chicken-and-egg problem. The controller
> needs to persist the UUID in the metadata topic in order to assign one
> successfully, but the metadata topic is needed to elect a controller
> (KIP-631). So, this probably needs a bit more thought.
>
> Jun
>
> On Thu, Sep 24, 2020 at 3:04 AM Ismael Juma <ism...@juma.me.uk> wrote:
>
> > Also, can we provide more details on how the Partition Metadata file will
> > be used?
> >
> > Ismael
> >
> > On Thu, Sep 24, 2020 at 3:01 AM Ismael Juma <ism...@juma.me.uk> wrote:
> >
> > > Hi Justine,
> > >
> > > I think we need to update the "Rejected Alternatives" section to take
> > into
> > > account that the proposal now removes the topic name from the fetch
> > > request. Also, if we are removing it from the Fetch request, does it
> make
> > > sense not to remove it from similar requests like ListOffsetRequest?
> > >
> > > Ismael
> > >
> > > On Thu, Sep 24, 2020 at 2:46 AM David Jacot <dja...@confluent.io>
> wrote:
> > >
> > >> Hi Justine,
> > >>
> > >> Thanks for the KIP. I finally had time to read it :). It is a great
> > >> improvement.
> > >>
> > >> I have a few comments/questions:
> > >>
> > >> 1. It seems that the schema of the StopReplicaRequest is slightly
> > >> outdated.
> > >> We
> > >> did some changes as part of KIP-570. V3 is already organized by
> topics.
> > >>
> > >> 2. I just want to make sure that I understand the reconciliation
> > >> logic correctly. When
> > >> an "INCREMENTAL" LeaderAndIsr Request is received, the broker will
> also
> > >> reconcile
> > >> when the local uuid does not match the uuid in the request, right? In
> > this
> > >> case, the
> > >> local log is staged for deletion.
> > >>
> > >> 3. In the documentation of the `delete.stale.topic.delay.ms` config,
> it
> > >> says "When a
> > >> FULL LeaderAndIsrRequest is received..." but I suppose it applies to
> > both
> > >> types.
> > >>
> > >> Best,
> > >> David
> > >>
> > >> On Thu, Sep 24, 2020 at 1:40 AM Justine Olshan <jols...@confluent.io>
> > >> wrote:
> > >>
> > >> > Hi Jun,
> > >> >
> > >> > Thanks for the comments. I apologize for some of the typos and
> > >> confusion.
> > >> > I’ve updated the KIP to fix some of the issues you mentioned.
> > >> >
> > >> > 20.2 I’ve changed the type to String.
> > >> > 20.1/3 I’ve updated the TopicZNode to fix formatting and reflect the
> > >> latest
> > >> > version before this change.
> > >> >
> > >> > 21. You are correct and I’ve removed this line. I’ve also added a
> line
> > >> > mentioning an IBP bump is necessary for migration
> > >> >
> > >> > 22. I think the wording was unclear but your summary is what was
> > >> intended
> > >> > by this line. I’ve updated to make this point more clear. “Remove
> > >> deleted
> > >> > topics from replicas by sending StopReplicaRequest V3 before the IBP
> > >> bump
> > >> > using the old logic, and using V4 and the new logic with topic IDs
> > after
> > >> > the IBP bump.”
> > >> >
> > >> > 23. I’ve removed the unspecified type from the KIP and mention that
> > IBP
> > >> > will be used to handle this request. “IBP will be used to determine
> > >> whether
> > >> > this new form of the request will be used. For older requests, we
> will
> > >> > ignore this field and default to previous behavior.”
> > >> >
> > >> > 24. I’ve fixed this typo.
> > >> >
> > >> > 25. I've added a topics at a higher level for LeaderAndIsrResponse
> v5,
> > >> > StopReplicaResponse v4. I've also changed StopReplicaRequest v4 to
> > have
> > >> > topics at a higher level.
> > >> >
> > >> > 26. I’ve updated forgotten_topics_data--added the topic ID and
> removed
> > >> the
> > >> > topic name
> > >> >
> > >> > 27. I’ve decided on plain text, and I’ve added an example.
> > >> >
> > >> > 28. I’ve added this idea to future work.
> > >> >
> > >> > Thanks again for taking a look,
> > >> >
> > >> > Justine
> > >> >
> > >> > On Wed, Sep 23, 2020 at 10:28 AM Jun Rao <j...@confluent.io> wrote:
> > >> >
> > >> > > Hi, Justine,
> > >> > >
> > >> > > Thanks for the response. Made another pass. A few more comments
> > below.
> > >> > >
> > >> > > 20. znode schema:
> > >> > > 20.1 It seems that {"name": "version", "type": "int", "id":
> "UUID",
> > >> > "doc":
> > >> > > "version id"} should be {"name": "version", "type": "int"},
> {"name":
> > >> > "id",
> > >> > > "type": "UUID", "doc": "version id"}.
> > >> > > 20.2 The znode format is JSON which doesn't have UUID type. So the
> > >> type
> > >> > > probably should be string?
> > >> > > 20.3 Also, the existing format used seems outdated. It should have
> > the
> > >> > > following format.
> > >> > > Json.encodeAsBytes(Map(
> > >> > > "version" -> 2,
> > >> > > "partitions" -> replicaAssignmentJson.asJava,
> > >> > > "adding_replicas" -> addingReplicasAssignmentJson.asJava,
> > >> > > "removing_replicas" -> removingReplicasAssignmentJson.asJava
> > >> > > ).asJava)
> > >> > > }
> > >> > >
> > >> > > 21. Migration: The KIP says "The migration process can take place
> > >> without
> > >> > > an inter-broker protocol bump, as the format stored in
> > >> > > /brokers/topics/[topic] will be compatible with older broker
> > >> versions."
> > >> > > However, since we are bumping up the version of inter-broker
> > >> requests, it
> > >> > > seems that we need to use IBP for migration.
> > >> > >
> > >> > > 22. The KIP says "Remove deleted topics from replicas by sending
> > >> > > StopReplicaRequest V3 for any topics which do not contain a topic
> > ID,
> > >> and
> > >> > > V4 for any topics which do contain a topic ID.". However, if we
> use
> > >> IBP,
> > >> > it
> > >> > > seems that the controller will either send StopReplicaRequest V3
> > >> > > or StopReplicaRequest V4, but never mixed V3 and V4 for different
> > >> topics.
> > >> > > Basically, before the IBP bump, V3 will be used. After the IBP
> bump,
> > >> > > topicId will be created and V4 will be used.
> > >> > >
> > >> > > 23. Given that we depend on IBP, do we still need "0 UNSPECIFIED"
> > >> > > in LeaderAndIsr?
> > >> > >
> > >> > > 24. LeaderAndIsrResponse v5 : It still has the topic field.
> > >> > >
> > >> > > 25. LeaderAndIsrResponse v5, StopReplicaResponse v4: Could we use
> > this
> > >> > > opportunity to organize the response in 2 levels, first by topic,
> > >> then by
> > >> > > partition, as most other requests/responses?
> > >> > >
> > >> > > 26. FetchRequest v13 : Should forgotten_topics_data use topicId
> too?
> > >> > >
> > >> > > 27. "This file can either be plain text (key/value pairs) or
> JSON."
> > >> Have
> > >> > we
> > >> > > decided which one to use? Also, it would be helpful to provide an
> > >> > example.
> > >> > >
> > >> > > 28. Future improvement: Another future improvement opportunity is
> to
> > >> use
> > >> > > topicId in GroupMetadataManager.offsetCommitKey in the
> offset_commit
> > >> > topic.
> > >> > > This may save some space.
> > >> > >
> > >> > > Thanks,
> > >> > >
> > >> > > Jun
> > >> > >
> > >> > > On Wed, Sep 23, 2020 at 8:50 AM Justine Olshan <
> > jols...@confluent.io>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Tom,
> > >> > > >
> > >> > > > Thanks for the comment. I think this is a really good idea and
> it
> > >> has
> > >> > > been
> > >> > > > added to the KIP under the newly added tooling section.
> > >> > > >
> > >> > > > Thanks again,
> > >> > > > Justine
> > >> > > >
> > >> > > > On Wed, Sep 23, 2020 at 3:17 AM Tom Bentley <
> tbent...@redhat.com>
> > >> > wrote:
> > >> > > >
> > >> > > > > Hi Justine,
> > >> > > > >
> > >> > > > > I know you started the vote thread, but on re-reading the KIP
> I
> > >> > noticed
> > >> > > > > that although the topic id is included in the MetadataResponse
> > >> it's
> > >> > not
> > >> > > > > surfaced in the output from `kafka-topics.sh --describe`.
> Maybe
> > >> that
> > >> > > was
> > >> > > > > intentional because ids are intentionally not really something
> > the
> > >> > user
> > >> > > > > should care deeply about, but it would also make life harder
> for
> > >> > anyone
> > >> > > > > debugging Kafka and this would likely get worse the more topic
> > ids
> > >> > got
> > >> > > > > rolled out across the protocols, clients etc. It seems likely
> > that
> > >> > > > > `kafka-topics.sh` will eventually need the ability to show the
> > id
> > >> of
> > >> > a
> > >> > > > > topic and perhaps find a topic name given an id. Is there any
> > >> reason
> > >> > > not
> > >> > > > to
> > >> > > > > implement that in this KIP?
> > >> > > > >
> > >> > > > > Many thanks,
> > >> > > > >
> > >> > > > > Tom
> > >> > > > >
> > >> > > > > On Mon, Sep 21, 2020 at 9:54 PM Justine Olshan <
> > >> jols...@confluent.io
> > >> > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Hi all,
> > >> > > > > >
> > >> > > > > > After thinking about it, I've decided to remove the topic
> name
> > >> from
> > >> > > the
> > >> > > > > > Fetch Request and Response after all. Since there are so
> many
> > of
> > >> > > these
> > >> > > > > > requests per second, it is worth removing the extra
> > information.
> > >> > I've
> > >> > > > > > updated the KIP to reflect this change.
> > >> > > > > >
> > >> > > > > > Please let me know if there is anything else we should
> discuss
> > >> > before
> > >> > > > > > voting.
> > >> > > > > >
> > >> > > > > > Thank you,
> > >> > > > > > Justine
> > >> > > > > >
> > >> > > > > > On Fri, Sep 18, 2020 at 9:46 AM Justine Olshan <
> > >> > jols...@confluent.io
> > >> > > >
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi Jun,
> > >> > > > > > >
> > >> > > > > > > I see what you are saying. For now we can remove the extra
> > >> > > > information.
> > >> > > > > > > I'll leave the option to add more fields to the file in
> the
> > >> > future.
> > >> > > > The
> > >> > > > > > KIP
> > >> > > > > > > has been updated to reflect this change.
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > Justine
> > >> > > > > > >
> > >> > > > > > > On Fri, Sep 18, 2020 at 8:46 AM Jun Rao <j...@confluent.io
> >
> > >> > wrote:
> > >> > > > > > >
> > >> > > > > > >> Hi, Justine,
> > >> > > > > > >>
> > >> > > > > > >> Thanks for the reply.
> > >> > > > > > >>
> > >> > > > > > >> 13. If the log directory is the source of truth, it means
> > >> that
> > >> > the
> > >> > > > > > >> redundant info in the metadata file will be ignored. Then
> > the
> > >> > > > question
> > >> > > > > > is
> > >> > > > > > >> why do we need to put the redundant info in the metadata
> > file
> > >> > now?
> > >> > > > > > >>
> > >> > > > > > >> Thanks,
> > >> > > > > > >>
> > >> > > > > > >> Jun
> > >> > > > > > >>
> > >> > > > > > >> On Thu, Sep 17, 2020 at 5:07 PM Justine Olshan <
> > >> > > > jols...@confluent.io>
> > >> > > > > > >> wrote:
> > >> > > > > > >>
> > >> > > > > > >> > Hi Jun,
> > >> > > > > > >> > Thanks for the quick response!
> > >> > > > > > >> >
> > >> > > > > > >> > 12. I've decided to bump up the versions on the
> requests
> > >> and
> > >> > > > updated
> > >> > > > > > the
> > >> > > > > > >> > KIP. I think it's good we thoroughly discussed the
> > options
> > >> > here,
> > >> > > > so
> > >> > > > > we
> > >> > > > > > >> know
> > >> > > > > > >> > we made a good choice. :)
> > >> > > > > > >> >
> > >> > > > > > >> > 13. This is an interesting situation. I think if this
> > does
> > >> > occur
> > >> > > > we
> > >> > > > > > >> should
> > >> > > > > > >> > give a warning. I agree that it's hard to know the
> source
> > >> of
> > >> > > truth
> > >> > > > > for
> > >> > > > > > >> sure
> > >> > > > > > >> > since the directory or the file could be manually
> > >> modified. I
> > >> > > > guess
> > >> > > > > > the
> > >> > > > > > >> > directory could be used as the source of truth. To be
> > >> honest,
> > >> > > I'm
> > >> > > > > not
> > >> > > > > > >> > really sure what happens in kafka when the log
> directory
> > is
> > >> > > > renamed
> > >> > > > > > >> > manually in such a way. I'm also wondering if the
> > >> situation is
> > >> > > > > > >> recoverable
> > >> > > > > > >> > in this scenario.
> > >> > > > > > >> >
> > >> > > > > > >> > Thanks,
> > >> > > > > > >> > Justine
> > >> > > > > > >> >
> > >> > > > > > >> > On Thu, Sep 17, 2020 at 4:28 PM Jun Rao <
> > j...@confluent.io>
> > >> > > wrote:
> > >> > > > > > >> >
> > >> > > > > > >> > > Hi, Justine,
> > >> > > > > > >> > >
> > >> > > > > > >> > > Thanks for the reply.
> > >> > > > > > >> > >
> > >> > > > > > >> > > 12. I don't have a strong preference either. However,
> > if
> > >> we
> > >> > > need
> > >> > > > > IBP
> > >> > > > > > >> > > anyway, maybe it's easier to just bump up the version
> > for
> > >> > all
> > >> > > > > inter
> > >> > > > > > >> > broker
> > >> > > > > > >> > > requests and add the topic id field as a regular
> > field. A
> > >> > > > regular
> > >> > > > > > >> field
> > >> > > > > > >> > is
> > >> > > > > > >> > > a bit more concise in wire transfer than a flexible
> > >> field.
> > >> > > > > > >> > >
> > >> > > > > > >> > > 13. The confusion that I was referring to is between
> > the
> > >> > topic
> > >> > > > > name
> > >> > > > > > >> and
> > >> > > > > > >> > > partition number between the log dir and the metadata
> > >> file.
> > >> > > For
> > >> > > > > > >> example,
> > >> > > > > > >> > if
> > >> > > > > > >> > > the log dir is topicA-1 and the metadata file in it
> has
> > >> > topicB
> > >> > > > and
> > >> > > > > > >> > > partition 0 (say due to a bug or manual
> modification),
> > >> which
> > >> > > one
> > >> > > > > do
> > >> > > > > > we
> > >> > > > > > >> > use
> > >> > > > > > >> > > as the source of truth?
> > >> > > > > > >> > >
> > >> > > > > > >> > > Jun
> > >> > > > > > >> > >
> > >> > > > > > >> > > On Thu, Sep 17, 2020 at 3:43 PM Justine Olshan <
> > >> > > > > > jols...@confluent.io>
> > >> > > > > > >> > > wrote:
> > >> > > > > > >> > >
> > >> > > > > > >> > > > Hi Jun,
> > >> > > > > > >> > > > Thanks for the comments.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > 12. I bumped the LeaderAndIsrRequest because I
> > removed
> > >> the
> > >> > > > topic
> > >> > > > > > >> name
> > >> > > > > > >> > > field
> > >> > > > > > >> > > > in the response. It may be possible to avoid
> bumping
> > >> the
> > >> > > > version
> > >> > > > > > >> > without
> > >> > > > > > >> > > > that change, but I may be missing something.
> > >> > > > > > >> > > > I believe StopReplica is actually on version 3 now,
> > but
> > >> > > > because
> > >> > > > > > >> > version 2
> > >> > > > > > >> > > > is flexible, I kept that listed as version 2 on the
> > KIP
> > >> > > page.
> > >> > > > > > >> However,
> > >> > > > > > >> > > you
> > >> > > > > > >> > > > may be right in that we may need to bump the
> version
> > on
> > >> > > > > > StopReplica
> > >> > > > > > >> to
> > >> > > > > > >> > > deal
> > >> > > > > > >> > > > with deletion differently as mentioned above. I
> don't
> > >> know
> > >> > > if
> > >> > > > I
> > >> > > > > > >> have a
> > >> > > > > > >> > > big
> > >> > > > > > >> > > > preference over used tagged fields or not.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > 13. I was thinking that in the case where the file
> > and
> > >> the
> > >> > > > > request
> > >> > > > > > >> > topic
> > >> > > > > > >> > > > ids don't match, it means that the broker's
> topic/the
> > >> one
> > >> > in
> > >> > > > the
> > >> > > > > > >> file
> > >> > > > > > >> > has
> > >> > > > > > >> > > > been deleted. In that case, we would need to delete
> > the
> > >> > old
> > >> > > > > topic
> > >> > > > > > >> and
> > >> > > > > > >> > > start
> > >> > > > > > >> > > > receiving the new version. If the topic name were
> to
> > >> > change,
> > >> > > > but
> > >> > > > > > the
> > >> > > > > > >> > ids
> > >> > > > > > >> > > > still matched, the file would also need to update.
> > Am I
> > >> > > > missing
> > >> > > > > a
> > >> > > > > > >> case
> > >> > > > > > >> > > > where the file would be correct and not the
> request?
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > Thanks,
> > >> > > > > > >> > > > Justine
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > On Thu, Sep 17, 2020 at 3:18 PM Jun Rao <
> > >> j...@confluent.io
> > >> > >
> > >> > > > > wrote:
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > > Hi, Justine,
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > Thanks for the reply. A couple of more comments
> > >> below.
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > 12. ListOffset and OffsetForLeader currently
> don't
> > >> > support
> > >> > > > > > >> flexible
> > >> > > > > > >> > > > fields.
> > >> > > > > > >> > > > > So, we have to bump up the version number and use
> > >> IBP at
> > >> > > > least
> > >> > > > > > for
> > >> > > > > > >> > > these
> > >> > > > > > >> > > > > two requests. Note that it seems 2.7.0 will
> require
> > >> IBP
> > >> > > > anyway
> > >> > > > > > >> > because
> > >> > > > > > >> > > of
> > >> > > > > > >> > > > > changes in KAFKA-10435. Also, it seems that the
> > >> version
> > >> > > for
> > >> > > > > > >> > > > > LeaderAndIsrRequest and StopReplica are bumped
> even
> > >> > though
> > >> > > > we
> > >> > > > > > only
> > >> > > > > > >> > > added
> > >> > > > > > >> > > > a
> > >> > > > > > >> > > > > tagged field. But since IBP is needed anyway, we
> > may
> > >> > want
> > >> > > to
> > >> > > > > > >> revisit
> > >> > > > > > >> > > the
> > >> > > > > > >> > > > > overall tagged field choice.
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > 13. The only downside is the potential confusion
> on
> > >> > which
> > >> > > > one
> > >> > > > > is
> > >> > > > > > >> the
> > >> > > > > > >> > > > source
> > >> > > > > > >> > > > > of truth if they don't match. Another option is
> to
> > >> > include
> > >> > > > > those
> > >> > > > > > >> > fields
> > >> > > > > > >> > > > in
> > >> > > > > > >> > > > > the metadata file when we actually change the
> > >> directory
> > >> > > > > > structure.
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > Thanks,
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > Jun
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > On Thu, Sep 17, 2020 at 2:01 PM Justine Olshan <
> > >> > > > > > >> jols...@confluent.io
> > >> > > > > > >> > >
> > >> > > > > > >> > > > > wrote:
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > > Hello all,
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > > I've thought some more about removing the topic
> > >> name
> > >> > > field
> > >> > > > > > from
> > >> > > > > > >> > some
> > >> > > > > > >> > > of
> > >> > > > > > >> > > > > the
> > >> > > > > > >> > > > > > requests. On closer inspection of the
> > >> > > requests/responses,
> > >> > > > it
> > >> > > > > > >> seems
> > >> > > > > > >> > > that
> > >> > > > > > >> > > > > the
> > >> > > > > > >> > > > > > internal changes would be much larger than I
> > >> expected.
> > >> > > > Some
> > >> > > > > > >> > protocols
> > >> > > > > > >> > > > > > involve clients, so they would require changes
> > too.
> > >> > I'm
> > >> > > > > > thinking
> > >> > > > > > >> > that
> > >> > > > > > >> > > > for
> > >> > > > > > >> > > > > > now, removing the topic name from these
> requests
> > >> and
> > >> > > > > responses
> > >> > > > > > >> are
> > >> > > > > > >> > > out
> > >> > > > > > >> > > > of
> > >> > > > > > >> > > > > > scope.
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > > I have decided to just keep the change
> > >> > > > LeaderAndIsrResponse
> > >> > > > > to
> > >> > > > > > >> > remove
> > >> > > > > > >> > > > the
> > >> > > > > > >> > > > > > topic name, and have updated the KIP to reflect
> > >> this
> > >> > > > > change. I
> > >> > > > > > >> have
> > >> > > > > > >> > > > also
> > >> > > > > > >> > > > > > mentioned the other requests and responses in
> > >> future
> > >> > > work.
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > > I'm hoping to start the voting process soon, so
> > >> let me
> > >> > > > know
> > >> > > > > if
> > >> > > > > > >> > there
> > >> > > > > > >> > > is
> > >> > > > > > >> > > > > > anything else we should discuss.
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > > Thank you,
> > >> > > > > > >> > > > > > Justine
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > > On Tue, Sep 15, 2020 at 3:57 PM Justine Olshan
> <
> > >> > > > > > >> > jols...@confluent.io
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > > > wrote:
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > > > Hello again,
> > >> > > > > > >> > > > > > > To follow up on some of the other comments:
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > 10/11) We can remove the topic name from
> these
> > >> > > > > > >> > requests/responses,
> > >> > > > > > >> > > > and
> > >> > > > > > >> > > > > > > that means we will just have to make a few
> > >> internal
> > >> > > > > changes
> > >> > > > > > to
> > >> > > > > > >> > make
> > >> > > > > > >> > > > > > > partitions accessible by topic id and
> > partition.
> > >> I
> > >> > can
> > >> > > > > > update
> > >> > > > > > >> the
> > >> > > > > > >> > > KIP
> > >> > > > > > >> > > > > to
> > >> > > > > > >> > > > > > > remove them unless anyone thinks they should
> > >> stay.
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > 12) Addressed in the previous email. I've
> > updated
> > >> > the
> > >> > > > KIP
> > >> > > > > to
> > >> > > > > > >> > > include
> > >> > > > > > >> > > > > > > tagged fields for the requests and responses.
> > >> (More
> > >> > on
> > >> > > > > that
> > >> > > > > > >> > below)
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > 13) I think part of the idea for including
> this
> > >> > > > > information
> > >> > > > > > >> is to
> > >> > > > > > >> > > > > prepare
> > >> > > > > > >> > > > > > > for future changes. Perhaps the directory
> > >> structure
> > >> > > > might
> > >> > > > > > >> change
> > >> > > > > > >> > > from
> > >> > > > > > >> > > > > > > topicName_partitionNumber to something like
> > >> > > > > > >> > > topicID_partitionNumber.
> > >> > > > > > >> > > > > Then
> > >> > > > > > >> > > > > > > it would be useful to have the topic name in
> > the
> > >> > file
> > >> > > > > since
> > >> > > > > > it
> > >> > > > > > >> > > would
> > >> > > > > > >> > > > > not
> > >> > > > > > >> > > > > > be
> > >> > > > > > >> > > > > > > in the directory structure. Supporting topic
> > >> renames
> > >> > > > might
> > >> > > > > > be
> > >> > > > > > >> > > easier
> > >> > > > > > >> > > > if
> > >> > > > > > >> > > > > > the
> > >> > > > > > >> > > > > > > other fields are included. Would there be any
> > >> > > downsides
> > >> > > > to
> > >> > > > > > >> > > including
> > >> > > > > > >> > > > > this
> > >> > > > > > >> > > > > > > information?
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > 14) Yes, we would need to copy the partition
> > >> > metadata
> > >> > > > > file
> > >> > > > > > in
> > >> > > > > > >> > this
> > >> > > > > > >> > > > > > > process. I've updated the KIP to include
> this.
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > 15) I believe Lucas meant v1 and v2 here. He
> > was
> > >> > > > referring
> > >> > > > > > to
> > >> > > > > > >> how
> > >> > > > > > >> > > the
> > >> > > > > > >> > > > > > > requests would fall under different IBP and
> > meant
> > >> > that
> > >> > > > > older
> > >> > > > > > >> > > brokers
> > >> > > > > > >> > > > > > would
> > >> > > > > > >> > > > > > > have to use the older version of the request
> > and
> > >> the
> > >> > > > > > existing
> > >> > > > > > >> > topic
> > >> > > > > > >> > > > > > > deletion process. At first, it seemed like
> > tagged
> > >> > > fields
> > >> > > > > > would
> > >> > > > > > >> > > > resolve
> > >> > > > > > >> > > > > > > the IBP issue. However, we may need IBP for
> > this
> > >> > > request
> > >> > > > > > after
> > >> > > > > > >> > all
> > >> > > > > > >> > > > > since
> > >> > > > > > >> > > > > > > the controller handles the topic deletion
> > >> > differently
> > >> > > > > > >> depending
> > >> > > > > > >> > on
> > >> > > > > > >> > > > the
> > >> > > > > > >> > > > > > IBP
> > >> > > > > > >> > > > > > > version. In an older version, we can't just
> > send
> > >> a
> > >> > > > > > StopReplica
> > >> > > > > > >> > > delete
> > >> > > > > > >> > > > > the
> > >> > > > > > >> > > > > > > topic immediately like we'd want to for this
> > KIP.
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > This makes me wonder if we want tagged fields
> > on
> > >> all
> > >> > > the
> > >> > > > > > >> requests
> > >> > > > > > >> > > > after
> > >> > > > > > >> > > > > > > all. Let me know your thoughts!
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > Justine
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > On Tue, Sep 15, 2020 at 1:03 PM Justine
> Olshan
> > <
> > >> > > > > > >> > > jols...@confluent.io
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > > > wrote:
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > >> Hi all,
> > >> > > > > > >> > > > > > >> Jun brought up a good point in his last
> email
> > >> about
> > >> > > > > tagged
> > >> > > > > > >> > fields,
> > >> > > > > > >> > > > and
> > >> > > > > > >> > > > > > >> I've updated the KIP to reflect that the
> > >> changes to
> > >> > > > > > requests
> > >> > > > > > >> and
> > >> > > > > > >> > > > > > responses
> > >> > > > > > >> > > > > > >> will be in the form of tagged fields to
> avoid
> > >> > > changing
> > >> > > > > IBP.
> > >> > > > > > >> > > > > > >>
> > >> > > > > > >> > > > > > >> Jun: I plan on sending a followup email to
> > >> address
> > >> > > some
> > >> > > > > of
> > >> > > > > > >> the
> > >> > > > > > >> > > other
> > >> > > > > > >> > > > > > >> points.
> > >> > > > > > >> > > > > > >>
> > >> > > > > > >> > > > > > >> Thanks,
> > >> > > > > > >> > > > > > >> Justine
> > >> > > > > > >> > > > > > >>
> > >> > > > > > >> > > > > > >> On Mon, Sep 14, 2020 at 4:25 PM Jun Rao <
> > >> > > > > j...@confluent.io>
> > >> > > > > > >> > wrote:
> > >> > > > > > >> > > > > > >>
> > >> > > > > > >> > > > > > >>> Hi, Justine,
> > >> > > > > > >> > > > > > >>>
> > >> > > > > > >> > > > > > >>> Thanks for the updated KIP. A few comments
> > >> below.
> > >> > > > > > >> > > > > > >>>
> > >> > > > > > >> > > > > > >>> 10. LeaderAndIsr Response: Do we need the
> > topic
> > >> > > name?
> > >> > > > > > >> > > > > > >>>
> > >> > > > > > >> > > > > > >>> 11. For the changed request/response, other
> > >> than
> > >> > > > > > >> LeaderAndIsr,
> > >> > > > > > >> > > > > > >>> UpdateMetadata, Metadata, do we need to
> > include
> > >> > the
> > >> > > > > topic
> > >> > > > > > >> name?
> > >> > > > > > >> > > > > > >>>
> > >> > > > > > >> > > > > > >>> 12. It seems that upgrades don't require
> IBP.
> > >> Does
> > >> > > > that
> > >> > > > > > mean
> > >> > > > > > >> > the
> > >> > > > > > >> > > > new
> > >> > > > > > >> > > > > > >>> fields
> > >> > > > > > >> > > > > > >>> in all the request/response are added as
> > tagged
> > >> > > fields
> > >> > > > > > >> without
> > >> > > > > > >> > > > > bumping
> > >> > > > > > >> > > > > > up
> > >> > > > > > >> > > > > > >>> the request version? It would be useful to
> > make
> > >> > that
> > >> > > > > > clear.
> > >> > > > > > >> > > > > > >>>
> > >> > > > > > >> > > > > > >>> 13. Partition Metadata file: Do we need to
> > >> include
> > >> > > the
> > >> > > > > > topic
> > >> > > > > > >> > name
> > >> > > > > > >> > > > and
> > >> > > > > > >> > > > > > the
> > >> > > > > > >> > > > > > >>> partition id since they are implied in the
> > >> > directory
> > >> > > > > name?
> > >> > > > > > >> > > > > > >>>
> > >> > > > > > >> > > > > > >>> 14. In the JBOD mode, we support moving a
> > >> > > partition's
> > >> > > > > data
> > >> > > > > > >> from
> > >> > > > > > >> > > one
> > >> > > > > > >> > > > > > disk
> > >> > > > > > >> > > > > > >>> to
> > >> > > > > > >> > > > > > >>> another. Will the new partition metadata
> file
> > >> be
> > >> > > > copied
> > >> > > > > > >> during
> > >> > > > > > >> > > that
> > >> > > > > > >> > > > > > >>> process?
> > >> > > > > > >> > > > > > >>>
> > >> > > > > > >> > > > > > >>> 15. The KIP says "Remove deleted topics
> from
> > >> > > replicas
> > >> > > > by
> > >> > > > > > >> > sending
> > >> > > > > > >> > > > > > >>> StopReplicaRequest V2 for any topics which
> do
> > >> not
> > >> > > > > contain
> > >> > > > > > a
> > >> > > > > > >> > topic
> > >> > > > > > >> > > > ID,
> > >> > > > > > >> > > > > > and
> > >> > > > > > >> > > > > > >>> V3 for any topics which do contain a topic
> > >> ID.".
> > >> > > > > However,
> > >> > > > > > it
> > >> > > > > > >> > > seems
> > >> > > > > > >> > > > > the
> > >> > > > > > >> > > > > > >>> updated controller will create all missing
> > >> topic
> > >> > IDs
> > >> > > > > first
> > >> > > > > > >> > before
> > >> > > > > > >> > > > > doing
> > >> > > > > > >> > > > > > >>> other actions. So, is StopReplicaRequest V2
> > >> > needed?
> > >> > > > > > >> > > > > > >>>
> > >> > > > > > >> > > > > > >>> Jun
> > >> > > > > > >> > > > > > >>>
> > >> > > > > > >> > > > > > >>> On Fri, Sep 11, 2020 at 10:31 AM John
> > Roesler <
> > >> > > > > > >> > > vvcep...@apache.org
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > > >>> wrote:
> > >> > > > > > >> > > > > > >>>
> > >> > > > > > >> > > > > > >>> > Thanks, Justine!
> > >> > > > > > >> > > > > > >>> >
> > >> > > > > > >> > > > > > >>> > Your response seems compelling to me.
> > >> > > > > > >> > > > > > >>> >
> > >> > > > > > >> > > > > > >>> > -John
> > >> > > > > > >> > > > > > >>> >
> > >> > > > > > >> > > > > > >>> > On Fri, 2020-09-11 at 10:17 -0700,
> Justine
> > >> > Olshan
> > >> > > > > wrote:
> > >> > > > > > >> > > > > > >>> > > Hello all,
> > >> > > > > > >> > > > > > >>> > > Thanks for continuing the discussion! I
> > >> have a
> > >> > > few
> > >> > > > > > >> > responses
> > >> > > > > > >> > > to
> > >> > > > > > >> > > > > > your
> > >> > > > > > >> > > > > > >>> > points.
> > >> > > > > > >> > > > > > >>> > >
> > >> > > > > > >> > > > > > >>> > > Tom: You are correct in that this KIP
> has
> > >> not
> > >> > > > > > mentioned
> > >> > > > > > >> the
> > >> > > > > > >> > > > > > >>> > > DeleteTopicsRequest. I think that this
> > >> would
> > >> > be
> > >> > > > out
> > >> > > > > of
> > >> > > > > > >> > scope
> > >> > > > > > >> > > > for
> > >> > > > > > >> > > > > > >>> now, but
> > >> > > > > > >> > > > > > >>> > > may be something worth adding in the
> > >> future.
> > >> > > > > > >> > > > > > >>> > >
> > >> > > > > > >> > > > > > >>> > > John: We did consider sequence ids, but
> > >> there
> > >> > > are
> > >> > > > a
> > >> > > > > > few
> > >> > > > > > >> > > reasons
> > >> > > > > > >> > > > > to
> > >> > > > > > >> > > > > > >>> favor
> > >> > > > > > >> > > > > > >>> > > UUIDs. There are several cases where
> > topics
> > >> > from
> > >> > > > > > >> different
> > >> > > > > > >> > > > > clusters
> > >> > > > > > >> > > > > > >>> may
> > >> > > > > > >> > > > > > >>> > > interact now and in the future. For
> > >> example,
> > >> > > > Mirror
> > >> > > > > > >> Maker 2
> > >> > > > > > >> > > may
> > >> > > > > > >> > > > > > >>> benefit
> > >> > > > > > >> > > > > > >>> > > from being able to detect when a
> cluster
> > >> being
> > >> > > > > > mirrored
> > >> > > > > > >> is
> > >> > > > > > >> > > > > deleted
> > >> > > > > > >> > > > > > >>> and
> > >> > > > > > >> > > > > > >>> > > recreated and globally unique
> identifiers
> > >> > would
> > >> > > > make
> > >> > > > > > >> > > resolving
> > >> > > > > > >> > > > > > issues
> > >> > > > > > >> > > > > > >>> > > easier than sequence IDs which may
> > collide
> > >> > > between
> > >> > > > > > >> > clusters.
> > >> > > > > > >> > > > > > KIP-405
> > >> > > > > > >> > > > > > >>> > > (tiered storage) will also benefit from
> > >> > globally
> > >> > > > > > unique
> > >> > > > > > >> IDs
> > >> > > > > > >> > > as
> > >> > > > > > >> > > > > > shared
> > >> > > > > > >> > > > > > >>> > > buckets may be used between clusters.
> > >> > > > > > >> > > > > > >>> > >
> > >> > > > > > >> > > > > > >>> > > Globally unique IDs would also make
> > >> > > functionality
> > >> > > > > like
> > >> > > > > > >> > moving
> > >> > > > > > >> > > > > > topics
> > >> > > > > > >> > > > > > >>> > > between disparate clusters easier in
> the
> > >> > future,
> > >> > > > > > >> simplify
> > >> > > > > > >> > any
> > >> > > > > > >> > > > > > future
> > >> > > > > > >> > > > > > >>> > > implementations of backups and
> restores,
> > >> and
> > >> > > more.
> > >> > > > > In
> > >> > > > > > >> > > general,
> > >> > > > > > >> > > > > > >>> unique IDs
> > >> > > > > > >> > > > > > >>> > > would ensure that the source cluster
> > >> topics do
> > >> > > not
> > >> > > > > > >> conflict
> > >> > > > > > >> > > > with
> > >> > > > > > >> > > > > > the
> > >> > > > > > >> > > > > > >>> > > destination cluster topics.
> > >> > > > > > >> > > > > > >>> > >
> > >> > > > > > >> > > > > > >>> > > If we were to use sequence ids, we
> would
> > >> need
> > >> > > > > > >> sufficiently
> > >> > > > > > >> > > > large
> > >> > > > > > >> > > > > > >>> cluster
> > >> > > > > > >> > > > > > >>> > > ids to be stored with the topic
> > >> identifiers or
> > >> > > we
> > >> > > > > run
> > >> > > > > > >> the
> > >> > > > > > >> > > risk
> > >> > > > > > >> > > > of
> > >> > > > > > >> > > > > > >>> > > collisions. This will give up any
> > >> advantage in
> > >> > > > > > >> compactness
> > >> > > > > > >> > > that
> > >> > > > > > >> > > > > > >>> sequence
> > >> > > > > > >> > > > > > >>> > > numbers may bring. Given these
> > advantages I
> > >> > > think
> > >> > > > it
> > >> > > > > > >> makes
> > >> > > > > > >> > > > sense
> > >> > > > > > >> > > > > to
> > >> > > > > > >> > > > > > >>> use
> > >> > > > > > >> > > > > > >>> > > UUIDs.
> > >> > > > > > >> > > > > > >>> > >
> > >> > > > > > >> > > > > > >>> > > Gokul: This is an interesting idea, but
> > >> this
> > >> > is
> > >> > > a
> > >> > > > > > >> breaking
> > >> > > > > > >> > > > > change.
> > >> > > > > > >> > > > > > >>> Out of
> > >> > > > > > >> > > > > > >>> > > scope for now, but maybe worth
> discussing
> > >> in
> > >> > the
> > >> > > > > > future.
> > >> > > > > > >> > > > > > >>> > >
> > >> > > > > > >> > > > > > >>> > > Hope this explains some of the
> decisions,
> > >> > > > > > >> > > > > > >>> > >
> > >> > > > > > >> > > > > > >>> > > Justine
> > >> > > > > > >> > > > > > >>> > >
> > >> > > > > > >> > > > > > >>> > >
> > >> > > > > > >> > > > > > >>> > >
> > >> > > > > > >> > > > > > >>> > > On Fri, Sep 11, 2020 at 8:27 AM Gokul
> > >> Ramanan
> > >> > > > > > >> Subramanian <
> > >> > > > > > >> > > > > > >>> > > gokul24...@gmail.com> wrote:
> > >> > > > > > >> > > > > > >>> > >
> > >> > > > > > >> > > > > > >>> > > > Hi.
> > >> > > > > > >> > > > > > >>> > > >
> > >> > > > > > >> > > > > > >>> > > > Thanks for the KIP.
> > >> > > > > > >> > > > > > >>> > > >
> > >> > > > > > >> > > > > > >>> > > > Have you thought about whether it
> makes
> > >> > sense
> > >> > > to
> > >> > > > > > >> support
> > >> > > > > > >> > > > > > >>> authorizing a
> > >> > > > > > >> > > > > > >>> > > > principal for a topic ID rather than
> a
> > >> topic
> > >> > > > name
> > >> > > > > to
> > >> > > > > > >> > > achieve
> > >> > > > > > >> > > > > > >>> tighter
> > >> > > > > > >> > > > > > >>> > > > security?
> > >> > > > > > >> > > > > > >>> > > >
> > >> > > > > > >> > > > > > >>> > > > Or is the topic ID fundamentally an
> > >> internal
> > >> > > > > detail
> > >> > > > > > >> > similar
> > >> > > > > > >> > > > to
> > >> > > > > > >> > > > > > >>> epochs
> > >> > > > > > >> > > > > > >>> > used
> > >> > > > > > >> > > > > > >>> > > > in a bunch of other places in Kafka?
> > >> > > > > > >> > > > > > >>> > > >
> > >> > > > > > >> > > > > > >>> > > > Thanks.
> > >> > > > > > >> > > > > > >>> > > >
> > >> > > > > > >> > > > > > >>> > > > On Fri, Sep 11, 2020 at 4:06 PM John
> > >> > Roesler <
> > >> > > > > > >> > > > > > vvcep...@apache.org>
> > >> > > > > > >> > > > > > >>> > wrote:
> > >> > > > > > >> > > > > > >>> > > >
> > >> > > > > > >> > > > > > >>> > > > > Hello Justine,
> > >> > > > > > >> > > > > > >>> > > > >
> > >> > > > > > >> > > > > > >>> > > > > Thanks for the KIP!
> > >> > > > > > >> > > > > > >>> > > > >
> > >> > > > > > >> > > > > > >>> > > > > I happen to have been confronted
> > >> recently
> > >> > > with
> > >> > > > > the
> > >> > > > > > >> need
> > >> > > > > > >> > > to
> > >> > > > > > >> > > > > keep
> > >> > > > > > >> > > > > > >>> > track of
> > >> > > > > > >> > > > > > >>> > > > a
> > >> > > > > > >> > > > > > >>> > > > > large number of topics as compactly
> > as
> > >> > > > > possible. I
> > >> > > > > > >> was
> > >> > > > > > >> > > > going
> > >> > > > > > >> > > > > to
> > >> > > > > > >> > > > > > >>> come
> > >> > > > > > >> > > > > > >>> > up
> > >> > > > > > >> > > > > > >>> > > > > with some way to dictionary encode
> > the
> > >> > topic
> > >> > > > > names
> > >> > > > > > >> as
> > >> > > > > > >> > > > > integers,
> > >> > > > > > >> > > > > > >>> but
> > >> > > > > > >> > > > > > >>> > this
> > >> > > > > > >> > > > > > >>> > > > > seems much better!
> > >> > > > > > >> > > > > > >>> > > > >
> > >> > > > > > >> > > > > > >>> > > > > Apologies if this has been raised
> > >> before,
> > >> > > but
> > >> > > > > I’m
> > >> > > > > > >> > > wondering
> > >> > > > > > >> > > > > > >>> about the
> > >> > > > > > >> > > > > > >>> > > > > choice of UUID vs sequence number
> for
> > >> the
> > >> > > ids.
> > >> > > > > > >> > Typically,
> > >> > > > > > >> > > > > I’ve
> > >> > > > > > >> > > > > > >>> seen
> > >> > > > > > >> > > > > > >>> > UUIDs
> > >> > > > > > >> > > > > > >>> > > > > in two situations:
> > >> > > > > > >> > > > > > >>> > > > > 1. When processes need to generate
> > >> > > > non-colliding
> > >> > > > > > >> > > > identifiers
> > >> > > > > > >> > > > > > >>> without
> > >> > > > > > >> > > > > > >>> > > > > coordination.
> > >> > > > > > >> > > > > > >>> > > > > 2. When the identifier needs to be
> > >> > > > “universally
> > >> > > > > > >> > unique”;
> > >> > > > > > >> > > > > I.e.,
> > >> > > > > > >> > > > > > >>> the
> > >> > > > > > >> > > > > > >>> > > > > identifier needs to distinguish the
> > >> entity
> > >> > > > from
> > >> > > > > > all
> > >> > > > > > >> > other
> > >> > > > > > >> > > > > > >>> entities
> > >> > > > > > >> > > > > > >>> > that
> > >> > > > > > >> > > > > > >>> > > > > could ever exist. This is useful in
> > >> cases
> > >> > > > where
> > >> > > > > > >> > entities
> > >> > > > > > >> > > > from
> > >> > > > > > >> > > > > > all
> > >> > > > > > >> > > > > > >>> > kinds
> > >> > > > > > >> > > > > > >>> > > > of
> > >> > > > > > >> > > > > > >>> > > > > systems get mixed together, such as
> > >> when
> > >> > > > dumping
> > >> > > > > > >> logs
> > >> > > > > > >> > > from
> > >> > > > > > >> > > > > all
> > >> > > > > > >> > > > > > >>> > processes
> > >> > > > > > >> > > > > > >>> > > > in
> > >> > > > > > >> > > > > > >>> > > > > a company into a common system.
> > >> > > > > > >> > > > > > >>> > > > >
> > >> > > > > > >> > > > > > >>> > > > > Maybe I’m being short-sighted, but
> it
> > >> > > doesn’t
> > >> > > > > seem
> > >> > > > > > >> like
> > >> > > > > > >> > > > > either
> > >> > > > > > >> > > > > > >>> really
> > >> > > > > > >> > > > > > >>> > > > > applies here. It seems like the
> > brokers
> > >> > > could
> > >> > > > > and
> > >> > > > > > >> would
> > >> > > > > > >> > > > > achieve
> > >> > > > > > >> > > > > > >>> > consensus
> > >> > > > > > >> > > > > > >>> > > > > when creating a topic anyway, which
> > is
> > >> all
> > >> > > > > that’s
> > >> > > > > > >> > > required
> > >> > > > > > >> > > > to
> > >> > > > > > >> > > > > > >>> > generate
> > >> > > > > > >> > > > > > >>> > > > > non-colliding sequence ids. For the
> > >> > second,
> > >> > > as
> > >> > > > > you
> > >> > > > > > >> > > mention,
> > >> > > > > > >> > > > > we
> > >> > > > > > >> > > > > > >>> could
> > >> > > > > > >> > > > > > >>> > > > assign
> > >> > > > > > >> > > > > > >>> > > > > a UUID to the cluster as a whole,
> > which
> > >> > > would
> > >> > > > > > render
> > >> > > > > > >> > any
> > >> > > > > > >> > > > > > resource
> > >> > > > > > >> > > > > > >>> > scoped
> > >> > > > > > >> > > > > > >>> > > > to
> > >> > > > > > >> > > > > > >>> > > > > the broker universally unique as
> > well.
> > >> > > > > > >> > > > > > >>> > > > >
> > >> > > > > > >> > > > > > >>> > > > > The reason I mention this is that,
> > >> > although
> > >> > > a
> > >> > > > > UUID
> > >> > > > > > >> is
> > >> > > > > > >> > way
> > >> > > > > > >> > > > > more
> > >> > > > > > >> > > > > > >>> > compact
> > >> > > > > > >> > > > > > >>> > > > > than topic names, it’s still 16
> > bytes.
> > >> In
> > >> > > > > > contrast,
> > >> > > > > > >> a
> > >> > > > > > >> > > > 4-byte
> > >> > > > > > >> > > > > > >>> integer
> > >> > > > > > >> > > > > > >>> > > > > sequence id would give us 4 billion
> > >> unique
> > >> > > > > topics
> > >> > > > > > >> per
> > >> > > > > > >> > > > > cluster,
> > >> > > > > > >> > > > > > >>> which
> > >> > > > > > >> > > > > > >>> > > > seems
> > >> > > > > > >> > > > > > >>> > > > > like enough ;)
> > >> > > > > > >> > > > > > >>> > > > >
> > >> > > > > > >> > > > > > >>> > > > > Considering the number of different
> > >> times
> > >> > > > these
> > >> > > > > > >> topic
> > >> > > > > > >> > > > > > >>> identifiers are
> > >> > > > > > >> > > > > > >>> > > > sent
> > >> > > > > > >> > > > > > >>> > > > > over the wire or stored in memory,
> it
> > >> > seems
> > >> > > > like
> > >> > > > > > it
> > >> > > > > > >> > might
> > >> > > > > > >> > > > be
> > >> > > > > > >> > > > > > >>> worth
> > >> > > > > > >> > > > > > >>> > the
> > >> > > > > > >> > > > > > >>> > > > > additional 4x space savings.
> > >> > > > > > >> > > > > > >>> > > > >
> > >> > > > > > >> > > > > > >>> > > > > What do you think about this?
> > >> > > > > > >> > > > > > >>> > > > >
> > >> > > > > > >> > > > > > >>> > > > > Thanks,
> > >> > > > > > >> > > > > > >>> > > > > John
> > >> > > > > > >> > > > > > >>> > > > >
> > >> > > > > > >> > > > > > >>> > > > > On Fri, Sep 11, 2020, at 03:20, Tom
> > >> > Bentley
> > >> > > > > wrote:
> > >> > > > > > >> > > > > > >>> > > > > > Hi Justine,
> > >> > > > > > >> > > > > > >>> > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > This looks like a very welcome
> > >> > > improvement.
> > >> > > > > > >> Thanks!
> > >> > > > > > >> > > > > > >>> > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > Maybe I missed it, but the KIP
> > >> doesn't
> > >> > > seem
> > >> > > > to
> > >> > > > > > >> > mention
> > >> > > > > > >> > > > > > changing
> > >> > > > > > >> > > > > > >>> > > > > > DeleteTopicsRequest to identify
> the
> > >> > topic
> > >> > > > > using
> > >> > > > > > an
> > >> > > > > > >> > id.
> > >> > > > > > >> > > > > Maybe
> > >> > > > > > >> > > > > > >>> > that's out
> > >> > > > > > >> > > > > > >>> > > > > of
> > >> > > > > > >> > > > > > >>> > > > > > scope, but DeleteTopicsRequest is
> > not
> > >> > > listed
> > >> > > > > > among
> > >> > > > > > >> > the
> > >> > > > > > >> > > > > Future
> > >> > > > > > >> > > > > > >>> Work
> > >> > > > > > >> > > > > > >>> > APIs
> > >> > > > > > >> > > > > > >>> > > > > > either.
> > >> > > > > > >> > > > > > >>> > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > Kind regards,
> > >> > > > > > >> > > > > > >>> > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > Tom
> > >> > > > > > >> > > > > > >>> > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > On Thu, Sep 10, 2020 at 3:59 PM
> > >> Satish
> > >> > > > > Duggana <
> > >> > > > > > >> > > > > > >>> > > > satish.dugg...@gmail.com
> > >> > > > > > >> > > > > > >>> > > > > > wrote:
> > >> > > > > > >> > > > > > >>> > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > Thanks Lucas/Justine for the
> nice
> > >> KIP.
> > >> > > > > > >> > > > > > >>> > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > It has several benefits which
> > also
> > >> > > include
> > >> > > > > > >> > > simplifying
> > >> > > > > > >> > > > > the
> > >> > > > > > >> > > > > > >>> topic
> > >> > > > > > >> > > > > > >>> > > > > > > deletion process by controller
> > and
> > >> > logs
> > >> > > > > > cleanup
> > >> > > > > > >> by
> > >> > > > > > >> > > > > brokers
> > >> > > > > > >> > > > > > in
> > >> > > > > > >> > > > > > >>> > corner
> > >> > > > > > >> > > > > > >>> > > > > > > cases.
> > >> > > > > > >> > > > > > >>> > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > Best,
> > >> > > > > > >> > > > > > >>> > > > > > > Satish.
> > >> > > > > > >> > > > > > >>> > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > On Wed, Sep 9, 2020 at 10:07 PM
> > >> > Justine
> > >> > > > > > Olshan <
> > >> > > > > > >> > > > > > >>> > jols...@confluent.io
> > >> > > > > > >> > > > > > >>> > > > > > > wrote:
> > >> > > > > > >> > > > > > >>> > > > > > > > Hello all, it's been almost a
> > >> year!
> > >> > > I've
> > >> > > > > > made
> > >> > > > > > >> > some
> > >> > > > > > >> > > > > > changes
> > >> > > > > > >> > > > > > >>> to
> > >> > > > > > >> > > > > > >>> > this
> > >> > > > > > >> > > > > > >>> > > > > KIP
> > >> > > > > > >> > > > > > >>> > > > > > > and hope to continue the
> > >> discussion.
> > >> > > > > > >> > > > > > >>> > > > > > > > One of the main changes I've
> > >> added
> > >> > is
> > >> > > > now
> > >> > > > > > the
> > >> > > > > > >> > > > metadata
> > >> > > > > > >> > > > > > >>> response
> > >> > > > > > >> > > > > > >>> > > > will
> > >> > > > > > >> > > > > > >>> > > > > > > include the topic ID (as Colin
> > >> > > suggested).
> > >> > > > > > >> Clients
> > >> > > > > > >> > > can
> > >> > > > > > >> > > > > > >>> obtain the
> > >> > > > > > >> > > > > > >>> > > > > topicID
> > >> > > > > > >> > > > > > >>> > > > > > > of a given topic through a
> > >> > > > TopicDescription.
> > >> > > > > > The
> > >> > > > > > >> > > > topicId
> > >> > > > > > >> > > > > > will
> > >> > > > > > >> > > > > > >>> > also be
> > >> > > > > > >> > > > > > >>> > > > > > > included with the
> UpdateMetadata
> > >> > > request.
> > >> > > > > > >> > > > > > >>> > > > > > > > Let me know what you all
> think.
> > >> > > > > > >> > > > > > >>> > > > > > > > Thank you,
> > >> > > > > > >> > > > > > >>> > > > > > > > Justine
> > >> > > > > > >> > > > > > >>> > > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > > On 2019/09/13 16:38:26,
> "Colin
> > >> > > McCabe" <
> > >> > > > > > >> > > > > > cmcc...@apache.org
> > >> > > > > > >> > > > > > >>> >
> > >> > > > > > >> > > > > > >>> > wrote:
> > >> > > > > > >> > > > > > >>> > > > > > > > > Hi Lucas,
> > >> > > > > > >> > > > > > >>> > > > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > > > Thanks for tackling this.
> > >> Topic
> > >> > IDs
> > >> > > > > are a
> > >> > > > > > >> > great
> > >> > > > > > >> > > > > idea,
> > >> > > > > > >> > > > > > >>> and
> > >> > > > > > >> > > > > > >>> > this
> > >> > > > > > >> > > > > > >>> > > > is
> > >> > > > > > >> > > > > > >>> > > > > a
> > >> > > > > > >> > > > > > >>> > > > > > > really good writeup.
> > >> > > > > > >> > > > > > >>> > > > > > > > > For
> /brokers/topics/[topic],
> > >> the
> > >> > > > schema
> > >> > > > > > >> version
> > >> > > > > > >> > > > > should
> > >> > > > > > >> > > > > > be
> > >> > > > > > >> > > > > > >>> > bumped
> > >> > > > > > >> > > > > > >>> > > > to
> > >> > > > > > >> > > > > > >>> > > > > > > version 3, rather than 2.
> > KIP-455
> > >> > > bumped
> > >> > > > > the
> > >> > > > > > >> > version
> > >> > > > > > >> > > > of
> > >> > > > > > >> > > > > > this
> > >> > > > > > >> > > > > > >>> > znode
> > >> > > > > > >> > > > > > >>> > > > to
> > >> > > > > > >> > > > > > >>> > > > > 2
> > >> > > > > > >> > > > > > >>> > > > > > > already :)
> > >> > > > > > >> > > > > > >>> > > > > > > > > Given that we're going to
> be
> > >> > seeing
> > >> > > > > these
> > >> > > > > > >> > things
> > >> > > > > > >> > > as
> > >> > > > > > >> > > > > > >>> strings
> > >> > > > > > >> > > > > > >>> > as
> > >> > > > > > >> > > > > > >>> > > > lot
> > >> > > > > > >> > > > > > >>> > > > > (in
> > >> > > > > > >> > > > > > >>> > > > > > > logs, in ZooKeeper, on the
> > >> > command-line,
> > >> > > > > > etc.),
> > >> > > > > > >> > does
> > >> > > > > > >> > > it
> > >> > > > > > >> > > > > > make
> > >> > > > > > >> > > > > > >>> > sense to
> > >> > > > > > >> > > > > > >>> > > > > use
> > >> > > > > > >> > > > > > >>> > > > > > > base64 when converting them to
> > >> > strings?
> > >> > > > > > >> > > > > > >>> > > > > > > > > Here is an example of the
> hex
> > >> > > > > > >> representation:
> > >> > > > > > >> > > > > > >>> > > > > > > > >
> > >> > 6fcb514b-b878-4c9d-95b7-8dc3a7ce6fd8
> > >> > > > > > >> > > > > > >>> > > > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > > > And here is an example in
> > >> base64.
> > >> > > > > > >> > > > > > >>> > > > > > > > > b8tRS7h4TJ2Vt43Dp85v2A
> > >> > > > > > >> > > > > > >>> > > > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > > > The base64 version saves 15
> > >> > letters
> > >> > > > (to
> > >> > > > > be
> > >> > > > > > >> > fair,
> > >> > > > > > >> > > 4
> > >> > > > > > >> > > > of
> > >> > > > > > >> > > > > > >>> those
> > >> > > > > > >> > > > > > >>> > were
> > >> > > > > > >> > > > > > >>> > > > > > > dashes that we could have
> elided
> > in
> > >> > the
> > >> > > > hex
> > >> > > > > > >> > > > > > representation.)
> > >> > > > > > >> > > > > > >>> > > > > > > > > Another thing to consider
> is
> > >> that
> > >> > we
> > >> > > > > > should
> > >> > > > > > >> > > specify
> > >> > > > > > >> > > > > > that
> > >> > > > > > >> > > > > > >>> the
> > >> > > > > > >> > > > > > >>> > > > > > > all-zeroes UUID is not a valid
> > >> topic
> > >> > > UUID.
> > >> > > > > > We
> > >> > > > > > >> > can't
> > >> > > > > > >> > > > use
> > >> > > > > > >> > > > > > >>> null
> > >> > > > > > >> > > > > > >>> > for
> > >> > > > > > >> > > > > > >>> > > > this
> > >> > > > > > >> > > > > > >>> > > > > > > because we can't pass a null
> UUID
> > >> over
> > >> > > the
> > >> > > > > RPC
> > >> > > > > > >> > > protocol
> > >> > > > > > >> > > > > > >>> (there
> > >> > > > > > >> > > > > > >>> > is no
> > >> > > > > > >> > > > > > >>> > > > > > > special pattern for null, nor
> do
> > we
> > >> > want
> > >> > > > to
> > >> > > > > > >> waste
> > >> > > > > > >> > > space
> > >> > > > > > >> > > > > > >>> reserving
> > >> > > > > > >> > > > > > >>> > > > such
> > >> > > > > > >> > > > > > >>> > > > > a
> > >> > > > > > >> > > > > > >>> > > > > > > pattern.)
> > >> > > > > > >> > > > > > >>> > > > > > > > > Maybe I missed it, but did
> > you
> > >> > > > describe
> > >> > > > > > >> > > "migration
> > >> > > > > > >> > > > > > of...
> > >> > > > > > >> > > > > > >>> > existing
> > >> > > > > > >> > > > > > >>> > > > > > > topic[s] without topic IDs" in
> > >> detail
> > >> > in
> > >> > > > any
> > >> > > > > > >> > section?
> > >> > > > > > >> > > > It
> > >> > > > > > >> > > > > > >>> seems
> > >> > > > > > >> > > > > > >>> > like
> > >> > > > > > >> > > > > > >>> > > > > when
> > >> > > > > > >> > > > > > >>> > > > > > > the new controller becomes
> > active,
> > >> it
> > >> > > > should
> > >> > > > > > >> just
> > >> > > > > > >> > > > > generate
> > >> > > > > > >> > > > > > >>> random
> > >> > > > > > >> > > > > > >>> > > > > UUIDs for
> > >> > > > > > >> > > > > > >>> > > > > > > these, and write the random
> UUIDs
> > >> back
> > >> > > to
> > >> > > > > > >> > ZooKeeper.
> > >> > > > > > >> > > > It
> > >> > > > > > >> > > > > > >>> would be
> > >> > > > > > >> > > > > > >>> > > > good
> > >> > > > > > >> > > > > > >>> > > > > to
> > >> > > > > > >> > > > > > >>> > > > > > > spell that out. We should make
> > it
> > >> > clear
> > >> > > > > that
> > >> > > > > > >> this
> > >> > > > > > >> > > > > happens
> > >> > > > > > >> > > > > > >>> > regardless
> > >> > > > > > >> > > > > > >>> > > > > of
> > >> > > > > > >> > > > > > >>> > > > > > > the inter-broker protocol
> version
> > >> > (it's
> > >> > > a
> > >> > > > > > >> > compatible
> > >> > > > > > >> > > > > > change).
> > >> > > > > > >> > > > > > >>> > > > > > > > > "LeaderAndIsrRequests
> > >> including an
> > >> > > > > > >> > > > is_every_partition
> > >> > > > > > >> > > > > > >>> flag"
> > >> > > > > > >> > > > > > >>> > > > seems a
> > >> > > > > > >> > > > > > >>> > > > > > > bit wordy. Can we just call
> > these
> > >> > "full
> > >> > > > > > >> > > > > > >>> LeaderAndIsrRequests"?
> > >> > > > > > >> > > > > > >>> > Then
> > >> > > > > > >> > > > > > >>> > > > > the
> > >> > > > > > >> > > > > > >>> > > > > > > RPC field could be named
> "full".
> > >> > Also,
> > >> > > it
> > >> > > > > > would
> > >> > > > > > >> > > > probably
> > >> > > > > > >> > > > > > be
> > >> > > > > > >> > > > > > >>> > better
> > >> > > > > > >> > > > > > >>> > > > > for the
> > >> > > > > > >> > > > > > >>> > > > > > > RPC field to be an enum of {
> > >> > > UNSPECIFIED,
> > >> > > > > > >> > > INCREMENTAL,
> > >> > > > > > >> > > > > FULL
> > >> > > > > > >> > > > > > >>> }, so
> > >> > > > > > >> > > > > > >>> > > > that
> > >> > > > > > >> > > > > > >>> > > > > we
> > >> > > > > > >> > > > > > >>> > > > > > > can cleanly handle old versions
> > (by
> > >> > > > treating
> > >> > > > > > >> them
> > >> > > > > > >> > as
> > >> > > > > > >> > > > > > >>> UNSPECIFIED)
> > >> > > > > > >> > > > > > >>> > > > > > > > > In the LeaderAndIsrRequest
> > >> > section,
> > >> > > > you
> > >> > > > > > >> write
> > >> > > > > > >> > "A
> > >> > > > > > >> > > > > final
> > >> > > > > > >> > > > > > >>> > deletion
> > >> > > > > > >> > > > > > >>> > > > > event
> > >> > > > > > >> > > > > > >>> > > > > > > will be secheduled for X ms
> after
> > >> the
> > >> > > > > > >> > > > LeaderAndIsrRequest
> > >> > > > > > >> > > > > > was
> > >> > > > > > >> > > > > > >>> > first
> > >> > > > > > >> > > > > > >>> > > > > > > received..." I guess the X
> was a
> > >> > > > > placeholder
> > >> > > > > > >> that
> > >> > > > > > >> > > you
> > >> > > > > > >> > > > > > >>> intended
> > >> > > > > > >> > > > > > >>> > to
> > >> > > > > > >> > > > > > >>> > > > > replace
> > >> > > > > > >> > > > > > >>> > > > > > > before posting? :) In any
> case,
> > >> this
> > >> > > > seems
> > >> > > > > > like
> > >> > > > > > >> > the
> > >> > > > > > >> > > > kind
> > >> > > > > > >> > > > > > of
> > >> > > > > > >> > > > > > >>> > thing
> > >> > > > > > >> > > > > > >>> > > > we'd
> > >> > > > > > >> > > > > > >>> > > > > > > want a configuration for.
> Let's
> > >> > > describe
> > >> > > > > that
> > >> > > > > > >> > > > > > configuration
> > >> > > > > > >> > > > > > >>> key
> > >> > > > > > >> > > > > > >>> > > > > somewhere
> > >> > > > > > >> > > > > > >>> > > > > > > in this KIP, including what its
> > >> > default
> > >> > > > > value
> > >> > > > > > >> is.
> > >> > > > > > >> > > > > > >>> > > > > > > > > We should probably also
> log a
> > >> > bunch
> > >> > > of
> > >> > > > > > >> messages
> > >> > > > > > >> > > at
> > >> > > > > > >> > > > > WARN
> > >> > > > > > >> > > > > > >>> level
> > >> > > > > > >> > > > > > >>> > > > when
> > >> > > > > > >> > > > > > >>> > > > > > > something is scheduled for
> > >> deletion,
> > >> > as
> > >> > > > > well.
> > >> > > > > > >> > (Maybe
> > >> > > > > > >> > > > > this
> > >> > > > > > >> > > > > > >>> was
> > >> > > > > > >> > > > > > >>> > > > > assumed, but
> > >> > > > > > >> > > > > > >>> > > > > > > it would be good to mention
> it).
> > >> > > > > > >> > > > > > >>> > > > > > > > > I feel like there are a few
> > >> > sections
> > >> > > > > that
> > >> > > > > > >> > should
> > >> > > > > > >> > > be
> > >> > > > > > >> > > > > > >>> moved to
> > >> > > > > > >> > > > > > >>> > > > > "rejected
> > >> > > > > > >> > > > > > >>> > > > > > > alternatives." For example, in
> > the
> > >> > > > > > DeleteTopics
> > >> > > > > > >> > > > section,
> > >> > > > > > >> > > > > > >>> since
> > >> > > > > > >> > > > > > >>> > we're
> > >> > > > > > >> > > > > > >>> > > > > not
> > >> > > > > > >> > > > > > >>> > > > > > > going to do option 1 or 2,
> these
> > >> > should
> > >> > > be
> > >> > > > > > moved
> > >> > > > > > >> > into
> > >> > > > > > >> > > > > > >>> "rejected
> > >> > > > > > >> > > > > > >>> > > > > > > alternatives," rather than
> > >> appearing
> > >> > > > > inline.
> > >> > > > > > >> > > Another
> > >> > > > > > >> > > > > case
> > >> > > > > > >> > > > > > >>> is
> > >> > > > > > >> > > > > > >>> > the
> > >> > > > > > >> > > > > > >>> > > > > "Should
> > >> > > > > > >> > > > > > >>> > > > > > > we remove topic name from the
> > >> protocol
> > >> > > > where
> > >> > > > > > >> > > possible"
> > >> > > > > > >> > > > > > >>> section.
> > >> > > > > > >> > > > > > >>> > This
> > >> > > > > > >> > > > > > >>> > > > > is
> > >> > > > > > >> > > > > > >>> > > > > > > clearly discussing a design
> > >> > alternative
> > >> > > > that
> > >> > > > > > >> we're
> > >> > > > > > >> > > not
> > >> > > > > > >> > > > > > >>> proposing
> > >> > > > > > >> > > > > > >>> > to
> > >> > > > > > >> > > > > > >>> > > > > > > implement: removing the topic
> > name
> > >> > from
> > >> > > > > those
> > >> > > > > > >> > > > protocols.
> > >> > > > > > >> > > > > > >>> > > > > > > > > Is it really necessary to
> > have
> > >> a
> > >> > new
> > >> > > > > > >> > > > > > >>> > /admin/delete_topics_by_id
> > >> > > > > > >> > > > > > >>> > > > > path
> > >> > > > > > >> > > > > > >>> > > > > > > in ZooKeeper? It seems like we
> > >> don't
> > >> > > > really
> > >> > > > > > >> need
> > >> > > > > > >> > > this.
> > >> > > > > > >> > > > > > >>> Whenever
> > >> > > > > > >> > > > > > >>> > > > > there is
> > >> > > > > > >> > > > > > >>> > > > > > > a new controller, we'll send
> out
> > >> full
> > >> > > > > > >> > > > > LeaderAndIsrRequests
> > >> > > > > > >> > > > > > >>> which
> > >> > > > > > >> > > > > > >>> > will
> > >> > > > > > >> > > > > > >>> > > > > > > trigger the stale topics to be
> > >> cleaned
> > >> > > up.
> > >> > > > > > The
> > >> > > > > > >> > > active
> > >> > > > > > >> > > > > > >>> > controller
> > >> > > > > > >> > > > > > >>> > > > will
> > >> > > > > > >> > > > > > >>> > > > > > > also send the full
> > >> LeaderAndIsrRequest
> > >> > > to
> > >> > > > > > >> brokers
> > >> > > > > > >> > > that
> > >> > > > > > >> > > > > are
> > >> > > > > > >> > > > > > >>> just
> > >> > > > > > >> > > > > > >>> > > > > starting
> > >> > > > > > >> > > > > > >>> > > > > > > up. So we don't really need
> > this
> > >> > kind
> > >> > > > of
> > >> > > > > > >> > two-phase
> > >> > > > > > >> > > > > > commit
> > >> > > > > > >> > > > > > >>> > (send
> > >> > > > > > >> > > > > > >>> > > > out
> > >> > > > > > >> > > > > > >>> > > > > > > StopReplicasRequest, get ACKs
> > from
> > >> all
> > >> > > > > nodes,
> > >> > > > > > >> > commit
> > >> > > > > > >> > > by
> > >> > > > > > >> > > > > > >>> removing
> > >> > > > > > >> > > > > > >>> > > > > > > /admin/delete_topics node) any
> > >> more.
> > >> > > > > > >> > > > > > >>> > > > > > > > > You mention that
> FetchRequest
> > >> will
> > >> > > now
> > >> > > > > > >> include
> > >> > > > > > >> > > UUID
> > >> > > > > > >> > > > > to
> > >> > > > > > >> > > > > > >>> avoid
> > >> > > > > > >> > > > > > >>> > > > issues
> > >> > > > > > >> > > > > > >>> > > > > > > where requests are made to
> stale
> > >> > > > partitions.
> > >> > > > > > >> > > However,
> > >> > > > > > >> > > > > > >>> adding a
> > >> > > > > > >> > > > > > >>> > UUID
> > >> > > > > > >> > > > > > >>> > > > to
> > >> > > > > > >> > > > > > >>> > > > > > > MetadataRequest is listed as
> > future
> > >> > > work,
> > >> > > > > out
> > >> > > > > > of
> > >> > > > > > >> > > scope
> > >> > > > > > >> > > > > for
> > >> > > > > > >> > > > > > >>> this
> > >> > > > > > >> > > > > > >>> > KIP.
> > >> > > > > > >> > > > > > >>> > > > > How
> > >> > > > > > >> > > > > > >>> > > > > > > will the client learn what the
> > >> topic
> > >> > > UUID
> > >> > > > > is,
> > >> > > > > > if
> > >> > > > > > >> > the
> > >> > > > > > >> > > > > > metadata
> > >> > > > > > >> > > > > > >>> > > > response
> > >> > > > > > >> > > > > > >>> > > > > > > doesn't include that
> information?
> > >> It
> > >> > > > seems
> > >> > > > > > like
> > >> > > > > > >> > > adding
> > >> > > > > > >> > > > > the
> > >> > > > > > >> > > > > > >>> UUID
> > >> > > > > > >> > > > > > >>> > to
> > >> > > > > > >> > > > > > >>> > > > > > > MetadataResponse would be an
> > >> > improvement
> > >> > > > > here
> > >> > > > > > >> that
> > >> > > > > > >> > > > might
> > >> > > > > > >> > > > > > not
> > >> > > > > > >> > > > > > >>> be
> > >> > > > > > >> > > > > > >>> > too
> > >> > > > > > >> > > > > > >>> > > > > hard to
> > >> > > > > > >> > > > > > >>> > > > > > > make.
> > >> > > > > > >> > > > > > >>> > > > > > > > > best,
> > >> > > > > > >> > > > > > >>> > > > > > > > > Colin
> > >> > > > > > >> > > > > > >>> > > > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > > > On Mon, Sep 9, 2019, at
> > 17:48,
> > >> > > Ryanne
> > >> > > > > > Dolan
> > >> > > > > > >> > > wrote:
> > >> > > > > > >> > > > > > >>> > > > > > > > > > Lucas, this would be
> great.
> > >> I've
> > >> > > run
> > >> > > > > > into
> > >> > > > > > >> > > issues
> > >> > > > > > >> > > > > with
> > >> > > > > > >> > > > > > >>> > topics
> > >> > > > > > >> > > > > > >>> > > > > being
> > >> > > > > > >> > > > > > >>> > > > > > > > > > resurrected accidentally,
> > >> since
> > >> > a
> > >> > > > > client
> > >> > > > > > >> > cannot
> > >> > > > > > >> > > > > > easily
> > >> > > > > > >> > > > > > >>> > > > > distinguish
> > >> > > > > > >> > > > > > >>> > > > > > > between
> > >> > > > > > >> > > > > > >>> > > > > > > > > > a deleted topic and a new
> > >> topic
> > >> > > with
> > >> > > > > the
> > >> > > > > > >> same
> > >> > > > > > >> > > > name.
> > >> > > > > > >> > > > > > I'd
> > >> > > > > > >> > > > > > >>> > need
> > >> > > > > > >> > > > > > >>> > > > the
> > >> > > > > > >> > > > > > >>> > > > > ID
> > >> > > > > > >> > > > > > >>> > > > > > > > > > accessible from the
> client
> > to
> > >> > > solve
> > >> > > > > that
> > >> > > > > > >> > issue,
> > >> > > > > > >> > > > but
> > >> > > > > > >> > > > > > >>> this
> > >> > > > > > >> > > > > > >>> > is a
> > >> > > > > > >> > > > > > >>> > > > > good
> > >> > > > > > >> > > > > > >>> > > > > > > first
> > >> > > > > > >> > > > > > >>> > > > > > > > > > step.
> > >> > > > > > >> > > > > > >>> > > > > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > > > > Ryanne
> > >> > > > > > >> > > > > > >>> > > > > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > > > > On Wed, Sep 4, 2019 at
> 1:41
> > >> PM
> > >> > > Lucas
> > >> > > > > > >> > > Bradstreet <
> > >> > > > > > >> > > > > > >>> > > > > lu...@confluent.io>
> > >> > > > > > >> > > > > > >>> > > > > > > wrote:
> > >> > > > > > >> > > > > > >>> > > > > > > > > > > Hi all,
> > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > > > > > I would like to kick
> off
> > >> > > > discussion
> > >> > > > > of
> > >> > > > > > >> > > KIP-516,
> > >> > > > > > >> > > > > an
> > >> > > > > > >> > > > > > >>> > > > > implementation
> > >> > > > > > >> > > > > > >>> > > > > > > of topic
> > >> > > > > > >> > > > > > >>> > > > > > > > > > > IDs for Kafka. Topic
> IDs
> > >> aim
> > >> > to
> > >> > > > > solve
> > >> > > > > > >> topic
> > >> > > > > > >> > > > > > >>> uniqueness
> > >> > > > > > >> > > > > > >>> > > > > problems in
> > >> > > > > > >> > > > > > >>> > > > > > > Kafka,
> > >> > > > > > >> > > > > > >>> > > > > > > > > > > where referring to a
> > topic
> > >> by
> > >> > > name
> > >> > > > > > >> alone is
> > >> > > > > > >> > > > > > >>> insufficient.
> > >> > > > > > >> > > > > > >>> > > > Such
> > >> > > > > > >> > > > > > >>> > > > > > > cases
> > >> > > > > > >> > > > > > >>> > > > > > > > > > > include when a topic
> has
> > >> been
> > >> > > > > deleted
> > >> > > > > > >> and
> > >> > > > > > >> > > > > recreated
> > >> > > > > > >> > > > > > >>> with
> > >> > > > > > >> > > > > > >>> > the
> > >> > > > > > >> > > > > > >>> > > > > same
> > >> > > > > > >> > > > > > >>> > > > > > > name.
> > >> > > > > > >> > > > > > >>> > > > > > > > > > > Unique identifiers will
> > >> help
> > >> > > > > simplify
> > >> > > > > > >> and
> > >> > > > > > >> > > > improve
> > >> > > > > > >> > > > > > >>> Kafka's
> > >> > > > > > >> > > > > > >>> > > > topic
> > >> > > > > > >> > > > > > >>> > > > > > > deletion
> > >> > > > > > >> > > > > > >>> > > > > > > > > > > process, as well as
> > prevent
> > >> > > cases
> > >> > > > > > where
> > >> > > > > > >> > > brokers
> > >> > > > > > >> > > > > may
> > >> > > > > > >> > > > > > >>> > > > incorrectly
> > >> > > > > > >> > > > > > >>> > > > > > > interact
> > >> > > > > > >> > > > > > >>> > > > > > > > > > > with stale versions of
> > >> topics.
> > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > >> > > > > > >> > > > > > >>> > > >
> > >> > > > > > >> > > > > > >>> >
> > >> > > > > > >> > > > > > >>>
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > >
> > >> > > > > > >> > >
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers
> > >> > > > > > >> > > > > > >>> > > > > > > > > > > Looking forward to your
> > >> > > thoughts.
> > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > >> > > > > > >> > > > > > >>> > > > > > > > > > > Lucas
> > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > >> > > > > > >> > > > > > >>> >
> > >> > > > > > >> > > > > > >>> >
> > >> > > > > > >> > > > > > >>>
> > >> > > > > > >> > > > > > >>
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > >
> > >> > > > > > >> > >
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
