Hi Jun,
Thanks for looking at it again. Here's the new spec. (I fixed the typo in
it too.)
{"name": "id", "type": "string", "doc": option[UUID]}
Justine
On Thu, Oct 1, 2020 at 5:03 PM Jun Rao wrote:
> Hi, Justine,
>
> Thanks for the update. The KIP looks good to me now. Just a minor comment
>
Hi, Justine,
Thanks for the update. The KIP looks good to me now. Just a minor comment
below.
30. Perhaps "option[UUID]" can be put in the doc.
Jun
On Thu, Oct 1, 2020 at 3:28 PM Justine Olshan wrote:
> Hi Jun,
> Thanks for the response!
>
> 30. I think I might have changed this in between. T
Hi Jun,
Thanks for the response!
30. I think I might have changed this in between. The current version
says: {"name":
"id", "type": "option[UUID]"}, "doc": topic id}
I have switched to the option type to cover the migration case where a
TopicZNode does not yet have a topic ID.
I understand that d
Hi, Justine,
Thanks for the summary. Just a few minor comments blow.
30. {"name": "id", "type": "string", "doc": "version id"}}: The doc should
say UUID. The issue seems unfixed.
40. Since UUID is public facing, could you include its definition?
41. StopReplicaResponse still includes the topic
Hello again,
I've taken some time to discuss some of the remaining points brought up by
the previous emails offline. Here are some of the conclusions.
1. Directory Structure:
There was some talk about whether the directory structure should be changed
to replace all topic names with topic IDs.
Thi
Hello all,
I just wanted to follow up on this discussion. Did we come to an
understanding about the directory structure?
I think the biggest question here is what is acceptable to leave out due to
scope vs. what is considered to be too much tech debt.
This KIP is already pretty large with the num
Hi Ismael,
If you do not store it in a metadata file or in the directory structure
would you then
require the LeaderAndIsrRequest following startup to give you some notion
of
topic name in memory? We would still need this information for the older
protocols, but
perhaps this is what's meant by tec
Hi Lucas,
Why would you include the name and id? I think you'd want to remove the
name from the directory name right? Jason's suggestion was that if you
remove the name from the directory, then why would you need the id name
mapping file?
Ismael
On Thu, Sep 24, 2020 at 4:24 PM Lucas Bradstreet
> 2. Part of the usage of the file is to have persistent storage of the
topic
ID and use it to compare with the ID supplied in the LeaderAndIsr Request.
There is some discussion in the KIP about changes to the directory
structure, but I believe directory changes were considered to be out of
scope w
Thanks Justine. Responses below:
> 1. Yes, the directory will still be based on the topic names.
LeaderAndIsrRequest is one of the few requests that will still contain the
topic name. So I think we have this covered. Sorry for confusion.
Ah, you're right. My eyes passed right over the field.
> 2
Hi Jason,
Thanks for your comments.
1. Yes, the directory will still be based on the topic names.
LeaderAndIsrRequest is one of the few requests that will still contain the
topic name. So I think we have this covered. Sorry for confusion.
2. Part of the usage of the file is to have persistent st
Hi Justine,
Thanks for picking up this work. I have a few questions/comments:
1. It sounds like the directory structure is still going to be based on
topic names. Do I have that right? One complication is that the
LeaderAndIsr request does not include the topic name any longer. This means
that a
Hi all,
Thanks for the discussion. I'm glad we are able to get our best ideas out
there.
David Jacot
1. I apologize for the incorrect information. I have fixed the KIP.
2. Yes. The difference between full and incremental is that on full we
check for the two types of stale request—topics on the br
Hi, Justine,
Thanks for the updated KIP. A few more comments below.
30. {"name": "id", "type": "string", "doc": "version id"}}: The doc should
say UUID.
31. LeaderAndIsrResponse v5 and StopReplicaResponse v4 : It seems there is
no need to add topic_id at partitions level.
32. Regarding partiti
Also, can we provide more details on how the Partition Metadata file will
be used?
Ismael
On Thu, Sep 24, 2020 at 3:01 AM Ismael Juma wrote:
> Hi Justine,
>
> I think we need to update the "Rejected Alternatives" section to take into
> account that the proposal now removes the topic name from t
Hi Justine,
I think we need to update the "Rejected Alternatives" section to take into
account that the proposal now removes the topic name from the fetch
request. Also, if we are removing it from the Fetch request, does it make
sense not to remove it from similar requests like ListOffsetRequest?
Hi Justine,
Thanks for the KIP. I finally had time to read it :). It is a great
improvement.
I have a few comments/questions:
1. It seems that the schema of the StopReplicaRequest is slightly outdated.
We
did some changes as part of KIP-570. V3 is already organized by topics.
2. I just want to
Hi Jun,
Thanks for the comments. I apologize for some of the typos and confusion.
I’ve updated the KIP to fix some of the issues you mentioned.
20.2 I’ve changed the type to String.
20.1/3 I’ve updated the TopicZNode to fix formatting and reflect the latest
version before this change.
21. You ar
Hi, Justine,
Thanks for the response. Made another pass. A few more comments below.
20. znode schema:
20.1 It seems that {"name": "version", "type": "int", "id": "UUID", "doc":
"version id"} should be {"name": "version", "type": "int"}, {"name": "id",
"type": "UUID", "doc": "version id"}.
20.2 Th
Hi Tom,
Thanks for the comment. I think this is a really good idea and it has been
added to the KIP under the newly added tooling section.
Thanks again,
Justine
On Wed, Sep 23, 2020 at 3:17 AM Tom Bentley wrote:
> Hi Justine,
>
> I know you started the vote thread, but on re-reading the KIP I
Hi Justine,
I know you started the vote thread, but on re-reading the KIP I noticed
that although the topic id is included in the MetadataResponse it's not
surfaced in the output from `kafka-topics.sh --describe`. Maybe that was
intentional because ids are intentionally not really something the us
Hi all,
After thinking about it, I've decided to remove the topic name from the
Fetch Request and Response after all. Since there are so many of these
requests per second, it is worth removing the extra information. I've
updated the KIP to reflect this change.
Please let me know if there is anyth
Hi Jun,
I see what you are saying. For now we can remove the extra information.
I'll leave the option to add more fields to the file in the future. The KIP
has been updated to reflect this change.
Thanks,
Justine
On Fri, Sep 18, 2020 at 8:46 AM Jun Rao wrote:
> Hi, Justine,
>
> Thanks for the
Hi, Justine,
Thanks for the reply.
13. If the log directory is the source of truth, it means that the
redundant info in the metadata file will be ignored. Then the question is
why do we need to put the redundant info in the metadata file now?
Thanks,
Jun
On Thu, Sep 17, 2020 at 5:07 PM Justine
Hi Jun,
Thanks for the quick response!
12. I've decided to bump up the versions on the requests and updated the
KIP. I think it's good we thoroughly discussed the options here, so we know
we made a good choice. :)
13. This is an interesting situation. I think if this does occur we should
give a w
Hi, Justine,
Thanks for the reply.
12. I don't have a strong preference either. However, if we need IBP
anyway, maybe it's easier to just bump up the version for all inter broker
requests and add the topic id field as a regular field. A regular field is
a bit more concise in wire transfer than a
Hi Jun,
Thanks for the comments.
12. I bumped the LeaderAndIsrRequest because I removed the topic name field
in the response. It may be possible to avoid bumping the version without
that change, but I may be missing something.
I believe StopReplica is actually on version 3 now, but because version
Hi, Justine,
Thanks for the reply. A couple of more comments below.
12. ListOffset and OffsetForLeader currently don't support flexible fields.
So, we have to bump up the version number and use IBP at least for these
two requests. Note that it seems 2.7.0 will require IBP anyway because of
change
Hello all,
I've thought some more about removing the topic name field from some of the
requests. On closer inspection of the requests/responses, it seems that the
internal changes would be much larger than I expected. Some protocols
involve clients, so they would require changes too. I'm thinking
Hello again,
To follow up on some of the other comments:
10/11) We can remove the topic name from these requests/responses, and that
means we will just have to make a few internal changes to make partitions
accessible by topic id and partition. I can update the KIP to remove them
unless anyone thi
Hi all,
Jun brought up a good point in his last email about tagged fields, and I've
updated the KIP to reflect that the changes to requests and responses will
be in the form of tagged fields to avoid changing IBP.
Jun: I plan on sending a followup email to address some of the other points.
Thanks
Hi, Justine,
Thanks for the updated KIP. A few comments below.
10. LeaderAndIsr Response: Do we need the topic name?
11. For the changed request/response, other than LeaderAndIsr,
UpdateMetadata, Metadata, do we need to include the topic name?
12. It seems that upgrades don't require IBP. Does
Thanks, Justine!
Your response seems compelling to me.
-John
On Fri, 2020-09-11 at 10:17 -0700, Justine Olshan wrote:
> Hello all,
> Thanks for continuing the discussion! I have a few responses to your points.
>
> Tom: You are correct in that this KIP has not mentioned the
> DeleteTopicsRequest
Hello all,
Thanks for continuing the discussion! I have a few responses to your points.
Tom: You are correct in that this KIP has not mentioned the
DeleteTopicsRequest. I think that this would be out of scope for now, but
may be something worth adding in the future.
John: We did consider sequence
Hi.
Thanks for the KIP.
Have you thought about whether it makes sense to support authorizing a
principal for a topic ID rather than a topic name to achieve tighter
security?
Or is the topic ID fundamentally an internal detail similar to epochs used
in a bunch of other places in Kafka?
Thanks.
Hello Justine,
Thanks for the KIP!
I happen to have been confronted recently with the need to keep track of a
large number of topics as compactly as possible. I was going to come up with
some way to dictionary encode the topic names as integers, but this seems much
better!
Apologies if this h
Hi Justine,
This looks like a very welcome improvement. Thanks!
Maybe I missed it, but the KIP doesn't seem to mention changing
DeleteTopicsRequest to identify the topic using an id. Maybe that's out of
scope, but DeleteTopicsRequest is not listed among the Future Work APIs
either.
Kind regards,
Thanks Lucas/Justine for the nice KIP.
It has several benefits which also include simplifying the topic
deletion process by controller and logs cleanup by brokers in corner
cases.
Best,
Satish.
On Wed, Sep 9, 2020 at 10:07 PM Justine Olshan wrote:
>
> Hello all, it's been almost a year! I've ma
Hello all, it's been almost a year! I've made some changes to this KIP and hope
to continue the discussion.
One of the main changes I've added is now the metadata response will include
the topic ID (as Colin suggested). Clients can obtain the topicID of a given
topic through a TopicDescription
Hi Lucas,
Thanks for tackling this. Topic IDs are a great idea, and this is a really
good writeup.
For /brokers/topics/[topic], the schema version should be bumped to version 3,
rather than 2. KIP-455 bumped the version of this znode to 2 already :)
Given that we're going to be seeing these
Lucas, this would be great. I've run into issues with topics being
resurrected accidentally, since a client cannot easily distinguish between
a deleted topic and a new topic with the same name. I'd need the ID
accessible from the client to solve that issue, but this is a good first
step.
Ryanne
O
Hi all,
I would like to kick off discussion of KIP-516, an implementation of topic
IDs for Kafka. Topic IDs aim to solve topic uniqueness problems in Kafka,
where referring to a topic by name alone is insufficient. Such cases
include when a topic has been deleted and recreated with the same name.
42 matches
Mail list logo
