Re: [DISCUSS] KIP-158 UPDATED: Enable source connectors to create new topics with specific configs in Kafka Connect during runtime

Thanks Konstantine. Looking forward to this feature. The KIP mentions: > For the *default* group this configuration is required. For any other group defined in topic.creation.groups this config is optional and if it's missing it gets the value the *default* group For the properties

Re: [DISCUSS] KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools

Comments below. On Thu, Jan 9, 2020 at 5:03 PM Colin McCabe wrote: > That's a good question. The current plan is for the 3.x releases to still > require ZooKeeper. What we will drop in 3.x is direct ZK access in > command-line administrative tools (unless those tools are specifically > about

Re: [VOTE] KIP-551: Expose disk read and write metrics

+1, LGTM. On Fri, Jan 10, 2020 at 2:19 PM Gwen Shapira wrote: > +1, thanks for driving this > > On Fri, Jan 10, 2020 at 2:17 PM Colin McCabe wrote: > > > > Hi all, > > > > I'd like to start the vote on KIP-551: Expose disk read and write > metrics. > > > > KIP:

Re: [DISCUSS] KIP-551: Expose disk read and write metrics

Thanks Colin, LGTM in general. The Linux documentation ( https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/proc.txt?id=HEAD#n1644) defines these metrics as read_bytes > -- > > I/O counter: bytes read > Attempt to count the number of bytes

Re: [DISCUSS] KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools

Thanks Colin, For the tools that only support zookeeper (zookeeper-security-migration.sh and zookeeper-shell.sh) should we be deprecating the entire tool for removal in a future 3.0 release? On Thu, Jan 9, 2020 at 4:24 PM Colin McCabe wrote: > Hi all, > > I wrote KIP about deprecating the

Re: [DISCUSS] KIP-589 Add API to Update Replica State in Controller

Hi David, Thanks for the KIP. > ReplicaStateEventResponse => ErrorCode [Topic [PartitionId]] >ErrorCode => Int32 >Topic => String >PartitionId => Int32 > ... > Partition-level errors: Based on my understanding of the response, it doesn't look like the controller has a way of

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

Hi Boyang, Thanks for the KIP. The KIP looks good. I have a few questions and comments. > As part of the KIP-500 initiative, we need to build a bridge release version of Kafka

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

Thanks for the KIP Cheng, > The default value will be 10 seconds. I think we should make the default the current behavior. Meaning the default should leverage the default connect timeout from the operating system. > Proposed Changes I don't fully understand this section. It seems like it is

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

t;> The NodeProvider is an AdminClient abstraction, right? Why wouldn't we > >> implement a connection setup timeout for all clients, not just AdminClient? > >> > >> best, > >> Colin > >> > >> On Mon, May 4, 2020, at 13:18, Colin McCabe wro

Re: [VOTE] KIP-589: Add API to update Replica state in Controller

+1. LGTM David! On Wed, May 20, 2020 at 12:22 PM David Arthur wrote: > > Hello, all. I'd like to start the vote for KIP-589 which proposes to add a > new AlterReplicaState RPC. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller > > Cheers,

Re: [VOTE] KIP-630: Kafka Raft Snapshot

Thanks everyone for the votes. KIP-630 has been accepted. Binding: Guozhang, Jason and Jun Non-binding: Ron and Lucas On Fri, Oct 9, 2020 at 4:33 PM Jose Garcia Sancio wrote: > > Thanks for the votes Jun, Jason, Ron, Lucas and Guozhang. > > Thanks for the feedback Ron and Ju

Re: [VOTE] KIP-630: Kafka Raft Snapshot

> ".new_records.size" must be written and ".change_records.ratio" must > > > be satisfied such that the number of changes (not accretions) divided > > > by N meets the ratio. > > > > > > Ron > > > > > > > > > > &g

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

Hi Jun and Colin, Some comments below. > 62.3 We added some configs in KIP-595 prefixed with "quorum" and we plan to > add some controller specific configs prefixed with "controller". KIP-630 > plans to add some other controller specific configs with no prefix. Should > we standardize all

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Thanks for the feedback Jason. I have made the following changes to the KIP: 1. Better explanation of how followers will manage snapshots and the replicated log. This includes the necessary changes when granting or requesting votes. 2. How the Snapshot's epoch will be used for the LastFetchEpoch

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Thanks for the detailed feedback Jun. The changes are here: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=25=24 Here is a summary of the change to the KIP: 1. Use end offset for snapshot and snapshot id. 2. Include default for all of the new configuration

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Thank you for the quick response Jun. Excuse the delayed response but I wanted to confirm some things regarding IBP. See comments below. Here are my changes to the KIP: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=30=28 > 40. LBO: Code wise, logStartOffset

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Thanks for the email Guozhang. > Thanks for the replies and the KIP updates. Just want to clarify one more > thing regarding my previous comment 3): I understand that when a snapshot > has completed loading, then we can use it in our handling logic of vote > request. And I understand that: > > 1)

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Comments below. Here are the change to the KIP: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=34=33 > 41. That's a good point. With compacted topic, the cleaning won't be done > until the active segment rolls. With snapshots, I guess we don't have this >

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Thanks Jason. Some comments below. > > Generally the number of snapshots on disk will be one. I suspect that > users will want some control over this. We can add a configuration > option that doesn't delete, or advances the log begin offset past, the > N latest snapshots. We can set the default

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Thanks for the reply Jun. Some comments below. Here are the changes: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=27=26 > 20. Good point on metadata cache. I think we need to make a decision > consistently. For example, if we decide that dedicated voter

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Hi Guozhang, Thanks for your feedback. It was very helpful. See my comments below. Changes to the KIP: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=28=27 On Sun, Sep 27, 2020 at 9:02 PM Guozhang Wang wrote: > > Hello Jose, > > Thanks for the KIP. Overall

[VOTE] KIP-630: Kafka Raft Snapshot

Hi all, I would like to start a vote on KIP-630. KIP: https://cwiki.apache.org/confluence/x/exV4CQ Discussion Thread: https://lists.apache.org/thread.html/r9468d1f276385695a2d6d48f6dfbdc504c445fc5745aaa606d138fed%40%3Cdev.kafka.apache.org%3E Thank you -- -Jose

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

I am going to propose that we take a vote on this KIP. Thank you Jason, Ron, Jun and Guozhang for the feedback and discussion. On Fri, Oct 2, 2020 at 3:11 PM Jose Garcia Sancio wrote: > > Thank you Jun! > > Changes: > https://cwiki.apache.org/confluence/pages/diffpagesbyversio

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Thank you Jun! Changes: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=37=36 > 41. Perhaps metadata.snapshot.min.records.size can just > be metadata.snapshot.min.records? Sounds good to me. Done. > 42. It's probably fine to change maxBytes for Fetch in a

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

I read through KIP-630 and made the following minor changes. https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=36=35 -- -Jose

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Thanks Ron for the additional comments and suggestions. Here are the changes to the KIP: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=17=15 On Wed, Jul 29, 2020 at 8:44 AM Ron Dagostino wrote: > > Thanks, Jose. It's looking good. Here is one minor

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

Thanks for the KIP Colin, Here is a partial review: 1. > Even when a broker and a controller are co-located in the same JVM, they must > have different node IDs Why? What problem are you trying to solve? 2. > Node IDs must be set in the configuration file for brokers and controllers. I

Re: [VOTE] KIP-595: A Raft Protocol for the Metadata Quorum

+1. Thanks for the detailed KIP! On Mon, Aug 3, 2020 at 11:03 AM Jason Gustafson wrote: > > Hi All, I'd like to start a vote on this proposal: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum. > The discussion has been active for a bit more

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

case > of __consumer_offsets, which sometimes has to support a very high rate of > offset commits, I think snapshotting would be a great tradeoff to reduce > load time on coordinator failover. The rate of change for metadata on the > other hand might not be as high, though it can be ver

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

Hi Unmesh, Very cool prototype! Hi Colin, The KIP proposes a record called IsrChange which includes the partition, topic, isr, leader and leader epoch. During normal operation ISR changes do not result in leader changes. Similarly, leader changes do not necessarily involve ISR changes. The

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Thanks for your feedback Jun. Here are my changes to the KIP: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=21=20 My comments are below... On Wed, Aug 5, 2020 at 1:59 PM Jun Rao wrote: > > Hi, Jose, > > Thanks for the KIP. A few comments blow. > > 10. I

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

Thanks for the KIP Ryan. Here are some of my observations. 1. > We will also be adding a new client configuration enable.dynamic.config to > both the producer and consumer that will be true by default so that the user > has the option to disable this feature. How about? "The Java producer and

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Once again, thanks for the feedback Jason, My changes to the KIP are here: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=18=17 And see my comments below... On Mon, Aug 3, 2020 at 1:57 PM Jason Gustafson wrote: > > Hi Jose, > > Thanks for the proposal. I

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

+1. Thanks for the KIP and looking forward to the improvement implementation. On Wed, Jun 17, 2020 at 2:24 PM Guozhang Wang wrote: > > Thanks for the KIP Boyang, +1 from me. > > > Guozhang > > On Wed, Jun 17, 2020 at 1:40 PM Colin McCabe wrote: > > > Thanks, Boyang! +1 (binding) > > > > best,

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Thanks Ron. Your comments and suggestions were helpful. You can see my changes to the KIP here: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=15=14 My comments are below... On Mon, Jul 27, 2020 at 11:29 AM Ron Dagostino wrote: > > Hi Jose. Thanks for the

[DISCUSS] KIP-630: Kafka Raft Snapshot

Hi All, I would like to start a discussion on KIP-630: https://cwiki.apache.org/confluence/x/exV4CQ This proposal specifies extensions to KIP-595 to support generating snapshots for the replicated log. Please let me know if you have any comments and suggestions. Thanks! -- -Jose