During the development of KIP-630 we made some minor changes to the
KIP to better match the implementation details. Here is a summary of
the changes we made to the KIP:
1. Added control records at the begin and end of the snapshots. The
control records are versioned. The snapshot header record
I am going to propose that we take a vote on this KIP.
Thank you Jason, Ron, Jun and Guozhang for the feedback and discussion.
On Fri, Oct 2, 2020 at 3:11 PM Jose Garcia Sancio wrote:
>
> Thank you Jun!
>
> Changes:
>
Thank you Jun!
Changes:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=37=36
> 41. Perhaps metadata.snapshot.min.records.size can just
> be metadata.snapshot.min.records?
Sounds good to me. Done.
> 42. It's probably fine to change maxBytes for Fetch in a
Hi, Jose,
Thanks for the updated KIP. Just a couple of minor comments.
41. Perhaps metadata.snapshot.min.records.size can just
be metadata.snapshot.min.records?
42. It's probably fine to change maxBytes for Fetch in a separate PR. I
brought this up since this KIP is changing the Fetch request.
I read through KIP-630 and made the following minor changes.
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=36=35
--
-Jose
Comments below.
Here are the change to the KIP:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=34=33
> 41. That's a good point. With compacted topic, the cleaning won't be done
> until the active segment rolls. With snapshots, I guess we don't have this
>
Thanks for the clarification Jose, that clears my confusions already :)
Guozhang
On Thu, Oct 1, 2020 at 10:51 AM Jose Garcia Sancio
wrote:
> Thanks for the email Guozhang.
>
> > Thanks for the replies and the KIP updates. Just want to clarify one more
> > thing regarding my previous comment
Hi, Jose,
Thanks for the reply. A few more comments.
41. That's a good point. With compacted topic, the cleaning won't be done
until the active segment rolls. With snapshots, I guess we don't have this
restriction? So, it would be useful to guard against too frequent
snapshotting. Does the new
Thanks for the email Guozhang.
> Thanks for the replies and the KIP updates. Just want to clarify one more
> thing regarding my previous comment 3): I understand that when a snapshot
> has completed loading, then we can use it in our handling logic of vote
> request. And I understand that:
>
> 1)
Thank you for the quick response Jun. Excuse the delayed response but
I wanted to confirm some things regarding IBP. See comments below.
Here are my changes to the KIP:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=30=28
> 40. LBO: Code wise, logStartOffset
Hello Jose,
Thanks for the replies and the KIP updates. Just want to clarify one more
thing regarding my previous comment 3): I understand that when a snapshot
has completed loading, then we can use it in our handling logic of vote
request. And I understand that:
1) Before a snapshot has been
Hi, Jose,
Thanks for the updated KIP. A few more comments below.
40. LBO: Code wise, logStartOffset is used in so many places like Log,
ReplicaManager, LogCleaner, ReplicaFetcher, checkpoint files, etc. I am not
if it's worth renaming in all those places. If the main concern is to
avoid
Hi Guozhang,
Thanks for your feedback. It was very helpful. See my comments below.
Changes to the KIP:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=28=27
On Sun, Sep 27, 2020 at 9:02 PM Guozhang Wang wrote:
>
> Hello Jose,
>
> Thanks for the KIP. Overall
Thanks for the reply Jun. Some comments below.
Here are the changes:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=27=26
> 20. Good point on metadata cache. I think we need to make a decision
> consistently. For example, if we decide that dedicated voter
Thanks Jason. Some comments below.
> > Generally the number of snapshots on disk will be one. I suspect that
> users will want some control over this. We can add a configuration
> option that doesn't delete, or advances the log begin offset past, the
> N latest snapshots. We can set the default
Hello Jose,
Thanks for the KIP. Overall it looks great. I have a few meta / minor
question, or maybe just clarifications below:
Meta:
1. I want to clarify that if only the active controller would generate
snapshots, OR would any voter in the quorum would generate snapshots, OR
would even
Hi, Jose,
Thanks for the reply. A few more comments below.
20. Good point on metadata cache. I think we need to make a decision
consistently. For example, if we decide that dedicated voter nodes don't
serve metadata requests, then we don't need to expose the voters host/port
to the client. Which
Thanks for the detailed feedback Jun.
The changes are here:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=25=24
Here is a summary of the change to the KIP:
1. Use end offset for snapshot and snapshot id.
2. Include default for all of the new configuration
Thanks Jose. Makes sense overall. A few specific responses below:
> Generally the number of snapshots on disk will be one. I suspect that
users will want some control over this. We can add a configuration
option that doesn't delete, or advances the log begin offset past, the
N latest snapshots.
Thanks for the feedback Jason.
I have made the following changes to the KIP:
1. Better explanation of how followers will manage snapshots and the
replicated log. This includes the necessary changes when granting or
requesting votes.
2. How the Snapshot's epoch will be used for the LastFetchEpoch
Hi, Jose,
Thanks for the updated KIP. A few more comments below.
20. "Metadata Cache: The component that generates snapshots, reads
snapshots and reads logs for observer replicas of the topic partition
__cluster_metadata." It seems this is needed on every broker, not just
observers?
21. Our
Hi Jose,
A few comments/questions below:
1. There is a comment in the proposal which suggests that we will maintain
multiple snapshots:
> Having multiple snapshots is useful for minimizing re-fetching of the
snapshot when a new snapshot is generated.
However, the document later says that
Thanks for your feedback Jun.
Here are my changes to the KIP:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=21=20
My comments are below...
On Wed, Aug 5, 2020 at 1:59 PM Jun Rao wrote:
>
> Hi, Jose,
>
> Thanks for the KIP. A few comments blow.
>
> 10. I
Hi, Jose,
Thanks for the KIP. A few comments blow.
10. I agree with Jason that it's useful to document the motivation a bit
clearer. Regarding semantic/performance, one benefit of snapshotting is
that it allows changes to be encoded incrementally instead of using the
full post image. For
Once again, thanks for the feedback Jason,
My changes to the KIP are here:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=18=17
And see my comments below...
On Mon, Aug 3, 2020 at 1:57 PM Jason Gustafson wrote:
>
> Hi Jose,
>
> Thanks for the proposal. I
Thanks for your feedback Jason. I'll have a more detailed reply and
update to the KIP by EOD today.
On Mon, Aug 3, 2020 at 1:57 PM Jason Gustafson wrote:
>
> Hi Jose,
>
> Thanks for the proposal. I think there are three main motivations for
> snapshotting over the existing compaction semantics.
Hi Jose,
Thanks for the proposal. I think there are three main motivations for
snapshotting over the existing compaction semantics.
First we are arguing that compaction is a poor semantic fit for how we want
to model the metadata in the cluster. We are trying to view the changes in
the cluster
Thanks Ron for the additional comments and suggestions.
Here are the changes to the KIP:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=17=15
On Wed, Jul 29, 2020 at 8:44 AM Ron Dagostino wrote:
>
> Thanks, Jose. It's looking good. Here is one minor
Thanks, Jose. It's looking good. Here is one minor correction:
<<< If the Kafka topic partition leader receives a fetch request with an
offset and epoch greater than or equal to the LBO (x + 1, a)
>>> If the Kafka topic partition leader receives a fetch request with an
offset and epoch greater
Thanks Ron. Your comments and suggestions were helpful. You can see my
changes to the KIP here:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=15=14
My comments are below...
On Mon, Jul 27, 2020 at 11:29 AM Ron Dagostino wrote:
>
> Hi Jose. Thanks for the
Hi Jose. Thanks for the KIP. Here are some questions and some nit corrections.
<<< In KIP-500 the Kafka Controller, which is the quorum leader from
KIP-595, will materialize the entries in the metadata log into memory.
Technically I think the quorum leader is referred to as the Active
31 matches
Mail list logo
