Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-30 Thread Colin McCabe
On Tue, Sep 29, 2020, at 17:43, Jason Gustafson wrote: > Hey Colin, > > Thanks for the hard work on this proposal. > > I'm gradually coming over to the idea of the controllers having separate > IDs. One of the benefits is that it allows us to separate the notion of > controller liveness from

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-28 Thread Colin McCabe
; No, broker.id does not need to be set in that case. best, Colin > > Thanks, > > Jun > > > > On Fri, Sep 25, 2020 at 2:17 PM Colin McCabe wrote: > > > On Fri, Sep 25, 2020, at 10:17, Jun Rao wrote: > > > Hi, Colin, > > > > > > Thanks

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-09-25 Thread Colin McCabe
like: > > > > They > > > > > > will > > > > > > > > >>>> get a > > > > > > > > >>>> random > > > > > > > > >>>> broker

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-25 Thread Colin McCabe
use broker.id? What will be missing? The controller has a separate ID from the broker, so knowing broker.id is not helpful here. best, Colin > > Thanks, > > Jun > > On Thu, Sep 24, 2020 at 10:55 PM Colin McCabe wrote: > > > On Thu, Sep 24, 2020, at 16:2

Re: [VOTE] KIP-584: Versioning scheme for features

2020-09-25 Thread Colin McCabe
t; > On Tue, Apr 28, 2020 at 11:24 PM Kowshik Prakasam > > wrote: > > > >> Hi all, > >> > >> This KIP vote has been open for ~12 days. The summary of the votes is > >> that we have 3 binding votes (Colin, Guozhang, Jun), and 3 non-binding > >

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-24 Thread Colin McCabe
adata catches up > to HWM. At that point, any local log not seen in the metadata can be > removed. Since the Fetch response returns the HWM, there seems to be enough > APIs to achieve this. > That's a very good point. I added a note about this under Broker Startup. best, Colin > &g

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-24 Thread Colin McCabe
led"? > I think it should show a list of all the storage directories, and whether each one is formatted. In addition, it should show whether kip.500.mode is enabled, and what the cluster id is. > > 56.2 Should the format mode take the config file as the input too like the &g

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-17 Thread Colin McCabe
kers to know about > it. > > Thanks, > Unmesh > > On Fri, Sep 11, 2020 at 10:32 PM Colin McCabe wrote: > > > Hi Unmesh, > > > > I think you're right that we should use a duration here rather than a > > time. As you said, the clock on the controller will probab

[VOTE] KIP-631: The Quorum-based Kafka Controller

2020-09-14 Thread Colin McCabe
Hi all, I'd like to call a vote on KIP-631: the quorum-based Kafka Controller. The KIP is here: https://cwiki.apache.org/confluence/x/4RV4CQ The DISCUSS thread is here: https://lists.apache.org/thread.html/r1ed098a88c489780016d963b065e8cb450a9080a4736457cd25f323c%40%3Cdev.kafka.apache.org%3E

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-11 Thread Colin McCabe
interval. Because registration.lease.timeout.ms, is configured > on the controller, the new active controller will extend all the leases by > registration.lease.timeout.ms. I see that it won't need last heartbeat > time. Agreed. best, Colin > > Thanks, > Unmesh > >

[jira] [Resolved] (KAFKA-10259) KIP-554: Add Broker-side SCRAM Config API

2020-09-04 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-10259. -- Fix Version/s: 2.7.0 Reviewer: Colin McCabe Resolution: Fixed > KIP-554:

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-09-04 Thread Colin McCabe
in the KIP. best, Colin > I assumed broker leases are implemented as a hard state. In that case, to > check for broker id conflict, we need to check the broker ids at two places > 1. Pending broker registrations (which are yet to be committed) 2. Already > committed broker registrations

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-09-02 Thread Colin McCabe
Hi Ron/Colin, > > > > > > > > > > Without any restrictions, if delegation tokens can be used to create > > > new > > > > > users or change the password of the user you are impersonating, you > > > also > > > > > get th

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-09-01 Thread Colin McCabe
wrote: > Hi Colin. I've removed that requirement from the KIP and updated the PR > accordingly. > > Ron > > On Fri, Aug 28, 2020 at 2:27 PM Colin McCabe wrote: > > > Hi Ron, > > > > Thanks for the update. I agree with all of these changes, except I think

[jira] [Resolved] (KAFKA-10384) Separate converters from generated messages

2020-08-31 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-10384. -- Fix Version/s: 2.7.0 Resolution: Fixed > Separate converters from generated messa

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-08-31 Thread Colin McCabe
I think it makes sense for the controller, since otherwise locking becomes very messy. I'm not sure I understand your question about duplicate broker ID detection, though. There's a section in the KIP about this -- is there a detail we should add there? best, Colin > > Thanks,

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-08-28 Thread Colin McCabe
f the broker." Can you give a little more context? best, Colin > > Thanks, > Unmesh > > On Sat, Aug 29, 2020 at 4:50 AM Colin McCabe wrote: > > > Hi all, > > > > I'm thinking of calling a vote on KIP-631 on Monday. Let me know if > > there's any more

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-08-28 Thread Colin McCabe
I will be adding more to the prototype and will be happy to help > and collaborate. > > Thanks, > Unmesh > > On Tue, Aug 11, 2020 at 12:28 AM Colin McCabe wrote: > > > Hi Jose, > > > > That'a s good point that I hadn't considered. It's probably worth h

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-08-28 Thread Colin McCabe
Hi Ron, Thanks for the update. I agree with all of these changes, except I think we should discuss this one further: On Wed, Aug 26, 2020, at 14:59, Ron Dagostino wrote: > > 2. We added a restriction to not allow users who authenticated using > delegation tokens to create or update user SCRAM

Re: [VOTE] KIP-595: A Raft Protocol for the Metadata Quorum

2020-08-16 Thread Colin McCabe
On Tue, Aug 11, 2020, at 11:30, Ismael Juma wrote: > Thanks for the KIP, +1 (binding). A couple of comments: > > 1. We have "quorum.voters=1@kafka-1:9092, 2@kafka-2:9092, > 3@kafka-3:9092". Could > this be a bit confusing given that the authority part of a url is defined > as "authority =

[jira] [Created] (KAFKA-10384) Separate converters from generated messages

2020-08-10 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-10384: Summary: Separate converters from generated messages Key: KAFKA-10384 URL: https://issues.apache.org/jira/browse/KAFKA-10384 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-08-10 Thread Colin McCabe
Hi Jose, That'a s good point that I hadn't considered. It's probably worth having a separate leader change message, as you mentioned. Hi Unmesh, Thanks, I'll take a look. best, Colin On Fri, Aug 7, 2020, at 11:56, Jose Garcia Sancio wrote: > Hi Unmesh, > > Very cool prototype! > > Hi

Re: [VOTE] KIP-595: A Raft Protocol for the Metadata Quorum

2020-08-04 Thread Colin McCabe
On Mon, Aug 3, 2020, at 20:55, Jason Gustafson wrote: > Hi Colin, > > Thanks for the responses. > > > I have a few lingering questions. I still don't like the fact that the > > leader epoch / fetch epoch is 31 bits. What happens when this rolls over? > > Can we just make this 63 bits now so

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-08-03 Thread Colin McCabe
know if you have any questions/concerns, > otherwise we can assume this change is acceptable. > > Ron > > On Tue, Jul 21, 2020 at 1:57 PM Colin McCabe wrote: > > > Hi all, > > > > With binding +1s from Rajini Sivaram, David Arthur, and Boyang Chen,

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-08-03 Thread Colin McCabe
On Mon, Aug 3, 2020, at 15:51, Jose Garcia Sancio wrote: > Thanks for the KIP Colin, > > Here is a partial review: > > > 1. Even when a broker and a controller are co-located in the same JVM, they > > must > > have different node IDs > > Why? What problem are you trying to solve? > Hi Jose,

Re: [VOTE] KIP-595: A Raft Protocol for the Metadata Quorum

2020-08-03 Thread Colin McCabe
Hi Jason, The KIP looks great. Thanks for all the work you've put into this. I have a few lingering questions. I still don't like the fact that the leader epoch / fetch epoch is 31 bits. What happens when this rolls over? Can we just make this 63 bits now so that we never have to worry

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-08-03 Thread Colin McCabe
metadata responses from the controller quorum anyway. best, Colin > Thanks, > Unmesh > > > > On Thu, Jul 30, 2020 at 6:04 AM Colin McCabe wrote: > > > On Thu, Jul 23, 2020, at 23:02, Boyang Chen wrote: > > > Hey Colin, > > > > > &

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-08-03 Thread Colin McCabe
to combine > > heartbeats and fetch and listed out the reason was due to extra complexity. > > However, we should also mention some cons caused by this model, for example > > we are doing 2X round trips to maintain a liveness, where as a regular > > follower it should always

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-29 Thread Colin McCabe
gone back and forth about, but overall I think it's good to at least implement the simple thing first. best, Colin > > Boyang > > On Wed, Jul 15, 2020 at 5:30 PM Colin McCabe wrote: > > > On Mon, Jul 13, 2020, at 11:08, Boyang Chen wrote: > > > Hey Colin, som

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-07-28 Thread Colin McCabe
ed points as well in the KIP, let me know > >> if you have further questions. > >> > >> Thanks, > >> Boyang > >> > >> On Tue, Jun 23, 2020 at 10:41 AM Ismael Juma wrote: > >> > >>> Option A is basically what I was thinking

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-07-28 Thread Colin McCabe
ell in the KIP, let me know > > >> if you have further questions. > > >> > > >> Thanks, > > >> Boyang > > >> > > >> On Tue, Jun 23, 2020 at 10:41 AM Ismael Juma wrote: > > >> > > >>>

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-21 Thread Colin McCabe
these requests are "upserts" (update if there, > otherwise insert). It was referred to as an "Addition" before, which I > felt was not technically correct. If you diff the most recent two versions > of the KIP it diffs pretty cleanly and makes the changes pretty apparent. >

[jira] [Resolved] (KAFKA-10174) Prefer --bootstrap-server ducktape tests using kafka_configs.sh

2020-07-16 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-10174. -- Fix Version/s: 2.7 Reviewer: Colin McCabe Resolution: Fixed > Pre

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-16 Thread Colin McCabe
cramCredentialDeletion > Yeah, there is a bit of a mismatch between "credentials" and "users." Really, these APIs are about credentials, not users. So I agree -- let's rename it. best, Colin > > On Wed, Jul 15, 2020 at 5:53 PM Colin McCabe wrote: > > &g

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

2020-07-15 Thread Colin McCabe
Hi Ying, Thanks for the response. It sounds like you agree that storing the metadata in the remote storage would be a better design overall. Given that that's true, is there any reason to include the worse implementation based on RocksDB? Choosing a long-term metadata store is not something

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-15 Thread Colin McCabe
but should `DeleteBrokerRecord` be > > `ShutdownBrokerRecord`? The broker is just getting removed from ISRs, but > > it would still be present in the replica set (I assume). > > > > Thanks, > > Jason > > > > On Sun, Jul 12, 2020 at 12:24 AM Colin McCabe wrote

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-15 Thread Colin McCabe
wers. Alternatives include fuzzy snapshots which can be done > > > concurrently. If this has been rejected, can you add some detail about > > why? > > > 4. More of a nit, but should `DeleteBrokerRecord` be > > > `ShutdownBrokerRecord`? The broker is just getting removed f

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-15 Thread Colin McCabe
Hi all, Thanks, everyone, for reviewing. Since we made a few changes to the RPCs in the last few days, I'm going to extend the vote until Monday and close it out then if it looks good. best, Colin On Wed, Jul 15, 2020, at 14:47, Colin McCabe wrote: > On Tue, Jul 14, 2020, at 16:23,

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-15 Thread Colin McCabe
ds? > Good call. I added this. best, Colin > > I haven't looked at the AlterScramUsers stuff yet; I'll look at that in > detail tomorrow. > > Ron > > > On Tue, Jul 14, 2020 at 4:11 PM Colin McCabe wrote: > > > On Tue, Jul 14, 2020, at 07:57, Ron Dagostino wr

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-14 Thread Colin McCabe
g? Thoughts? > > > > Also, separately, should the responses include a ThrottleTimeMs field? I > > believe so but would like to confirm. > > > > Ron > > > > On Mon, Jul 13, 2020 at 3:44 PM David Arthur wrote: > > > >> Thanks for t

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-13 Thread Colin McCabe
lterScramUsersRequest", > Other than that, +1 (binding) from me. > > > On Mon, Jul 13, 2020 at 8:38 AM Colin McCabe wrote: > > > Hi David, > > > > The API is for clients. Brokers will still listen to ZooKeeper to load > > the SCRAM informati

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-13 Thread Colin McCabe
d info through the RPC, how will brokers > load this info? (I'm presuming that they need it to configure > authentication) > > -David > > On Mon, Jul 13, 2020 at 10:57 AM Colin McCabe wrote: > > > On Fri, Jul 10, 2020, at 10:55, Boyang Chen wrote: > > > Hey Coli

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-13 Thread Colin McCabe
the SCRAM requests correct? > Yes, the client generates the salt before sending the request. best, Colin > Best, > Boyang > > On Fri, Jul 10, 2020 at 8:10 AM Rajini Sivaram > wrote: > > > +1 (binding) > > > > Thanks for the KIP, Colin! > > &g

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-12 Thread Colin McCabe
browse/ZOOKEEPER-965> > > Thanks, > Unmesh > > On Fri, Jul 10, 2020 at 11:32 AM Colin McCabe wrote: > > > Hi Unmesh, > > > > Yes, once the last stable offset advanced, we would consider the topic > > creation to be done, and then we could return succes

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-12 Thread Colin McCabe
ther always, and in no situation we can have > > TopicRecord without PartitionRecord. Not sure if there are other situations > > where multi-operation is needed. > > <https://issues.apache.org/jira/browse/ZOOKEEPER-965> > > > > Thanks, > > Unmesh > >

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

2020-07-10 Thread Colin McCabe
Hi all, Thanks for the KIP. I took a look and one thing that stood out to me is that the more metadata we have, the more storage we will need on local disk for the rocksDB database. This seems like it contradicts some of the goals of the project. Ideally the space we need on local disk

Re: [DISCUSS] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

2020-07-10 Thread Colin McCabe
is what most tools care about anyway. best, Colin On Thu, Jul 9, 2020, at 23:04, Colin McCabe wrote: > Yeah. The issue with subclassing is that it's a source compatibility > break, although not (I think) a binary compatibility break. I don't > think it's really worth it given that

Re: [DISCUSS] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

2020-07-10 Thread Colin McCabe
ribeLogDirsResult#[all, values] from > > old ones to the new ones, with a notice. Since we already deprecated > > the > > old ones, most users would already be moved into the new ones. So it > > does > > not occur any problems. > > 2.

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-10 Thread Colin McCabe
Previous HighWaterMark >CreateTopic || >|| >|| > > > > > > > > > > > > On Fri, Jul 10, 2020 at 1:30 AM Colin McCabe wrote: > > > On Thu, Jul 9, 2020, at 04:37, Unmesh Joshi wrote: > > > I see that, when a new topic is created, two metadata re

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-09 Thread Colin McCabe
to > > > the active controller rather than just the controller? > > > > > > "*Therefore, the controller must not make this future state "visible" to > > > the rest of the cluster until it has been made persistent – that is, > > until > &

[jira] [Resolved] (KAFKA-10256) Create a server gradle module for Java code needed only by servers

2020-07-09 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-10256. -- Resolution: Won't Fix > Create a server gradle module for Java code needed only by serv

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-09 Thread Colin McCabe
> Why? Is it necessary to state this? It seems like it might be an > > implementation detail rather than a necessary constraint/requirement that > > we declare publicly and would have to abide by. > > > > "*It will reject brokers whose metadata is too stale*". Wh

[VOTE] KIP-554: Add Broker-side SCRAM Config API

2020-07-09 Thread Colin McCabe
Hi all, I'd like to call a vote for KIP-554: Add a broker-side SCRAM configuration API. The KIP is here: https://cwiki.apache.org/confluence/x/ihERCQ The previous discussion thread is here:

[jira] [Created] (KAFKA-10256) Create a server gradle module for Java code needed only by servers

2020-07-09 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-10256: Summary: Create a server gradle module for Java code needed only by servers Key: KAFKA-10256 URL: https://issues.apache.org/jira/browse/KAFKA-10256 Project: Kafka

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-08 Thread Colin McCabe
ontext in KIP-500. Stale metadata is a general problem since it can confuse clients-- they don't see a topic which actually does exist, for example, or do see one that was deleted. > > "*it may lose subsequent conflicts if its broker epoch is stale*" This is > the first time a &quo

Re: [DISCUSS] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

2020-07-08 Thread Colin McCabe
Hi Dongjin, Hmm. I'm not sure I follow. How does deprecating DescribeLogDirsResponse.LogDirInfo help here? The issue is not so much the class, but the fact that it's exposed as a public API. So it seems appropriate to deprecate the methods that return it, but not the class itself, since

[DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-07 Thread Colin McCabe
Hi all, I posted a KIP about how the quorum-based controller envisioned in KIP-500 will work. Please take a look here: https://cwiki.apache.org/confluence/x/4RV4CQ best, Colin

Re: [DISCUSS] KIP-363

2020-07-07 Thread Colin McCabe
Hi Tom, Thanks for this. I think the tough part is probably the few messages that are still using manual serialization, which can't be easily converted to using this. So we will probably have to upgrade them to using automatic generation, or accept a little inconsistency for a while until

Re: [VOTE] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

2020-07-07 Thread Colin McCabe
Thanks, Tom. I tried to think of a better way to do this, but I think you're right that we probably just need different methods. +1 (binding). best, Colin On Mon, Jul 6, 2020, at 01:14, Tom Bentley wrote: > Hi, > > I'd like to start a vote on KIP-621 which is about deprecating methods in >

Re: [DISCUSS] Include min.insync.replicas in MetadataResponse to make Producer smarter in partitioning events

2020-07-07 Thread Colin McCabe
Hi Arvin, Thanks for the KIP. Unfortunately, I don't think this makes sense since it would increase the amount of data we send back in the metadata response, which is pretty bad for scalability. In general we probably want to avoid the case where we don't have the appropriate number of

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-06-30 Thread Colin McCabe
Hi Rajini, OK. Let's remove the encrypted credentials from ListScramUsersResponse and the associated API. I have updated the KIP-- take a look when you get a chance. best, Colin On Fri, May 15, 2020, at 06:54, Rajini Sivaram wrote: > Hi Colin, > > We have used different approaches for

Re: Running system tests on mac

2020-06-30 Thread Colin McCabe
Ducktape runs on Python 2. You can't use it with Python 3, as you are trying to do here. If anyone's interested in porting it to Python 3 it would be a good change. Otherwise, using docker as suggested here seems to be the best way to go. best, Colin On Mon, Jun 29, 2020, at 02:14, Gokul

Re: [VOTE] KIP-627: Expose Trogdor-specific JMX Metrics for Tasks and Agents

2020-06-25 Thread Colin McCabe
+1 (binding). Thanks, Sam. best, Colin On Thu, Jun 25, 2020, at 18:05, Gwen Shapira wrote: > +1 (binding) > > Thank you, Sam. It is great to see Trogdor getting the care it deserves. > > On Mon, Jun 22, 2020, 1:46 PM Sam Pal wrote: > > > Hi all, > > > > I would like to start a vote for

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-06-23 Thread Colin McCabe
> > > On Fri, Jun 19, 2020 at 3:18 PM Ismael Juma wrote: > > > > > > > Hi Colin, > > > > > > > > The KIP states in the Compatibility section (not Future work): > > > > > > > > "To support the proxy of requests, we need to build a channel for > > > > brokers to talk directly to the controller.

Re: [ANNOUNCE] New committer: Boyang Chen

2020-06-22 Thread Colin McCabe
Congratulations, Boyang! cheers, Colin On Mon, Jun 22, 2020, at 16:26, Guozhang Wang wrote: > The PMC for Apache Kafka has invited Boyang Chen as a committer and we are > pleased to announce that he has accepted! > > Boyang has been active in the Kafka community more than two years ago. > Since

Re: [DISCUSS]: KIP-625: Richer encodings for integral-typed protocol fields.

2020-06-19 Thread Colin McCabe
On Tue, Jun 16, 2020, at 07:42, Tom Bentley wrote: > Hi Colin, > > Thanks for taking a look at it. Replies inline... > > On Mon, Jun 15, 2020 at 6:22 PM Colin McCabe wrote: > > > Hi Tom, > > > > It's an interesting idea. Obviously protocol buffers

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-06-19 Thread Colin McCabe
gt; jsan...@confluent.io> > > > > wrote: > > > > > > > > > +1. > > > > > > > > > > Thanks for the KIP and looking forward to the improvement > > > implementation. > > > > > > > > > > On Wed, Jun 17

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-06-17 Thread Colin McCabe
Thanks, Boyang! +1 (binding) best, Colin On Mon, Jun 15, 2020, at 12:59, Boyang Chen wrote: > Thanks for more feedback Colin! I have addressed them in the KIP. > > Boyang > > On Mon, Jun 15, 2020 at 11:29 AM Colin McCabe wrote: > > > On Fri, Jun 12, 2020, at

Re: [DISCUSS] KIP-627: Expose Trogdor-specific JMX Metrics for Tasks and Agents

2020-06-17 Thread Colin McCabe
Hi Sam, Thanks for the KIP. Can you add some text clarifying whether a done task continues to be counted in the created-task-count? Looks good aside from that. best, Colin On Wed, Jun 17, 2020, at 12:31, Sam Pal wrote: > Hi all, > > I’d like to start a discussion about adding metrics to

Re: Broker side round robin on topic partitions when receiving messages

2020-06-15 Thread Colin McCabe
to decide the partition. > We noticed enqueuing and timeouts while several consumers were idle - which > made us take a better look on how the load is balanced. > > I believe the only way to perform equal load balance without having to know > other producers would be to do it on the Broke

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-06-15 Thread Colin McCabe
On Fri, Jun 12, 2020, at 15:30, Boyang Chen wrote: > Thanks Colin for the suggestions! > > On Fri, Jun 12, 2020 at 2:40 PM Colin McCabe wrote: > > > Hi Boyang, > > > > Thanks for the KIP! I think it's getting close. > > > > > For older requests th

Re: [DISCUSSION] KIP-619: Add internal topic creation support

2020-06-15 Thread Colin McCabe
Hi Cheng, The link from the main KIP page is an "edit link" meaning that it drops you into the editor for the wiki page. I think the link you meant to use is a "view link" that will just take you to view the page. In general I'm not sure what I'm supposed to take away from the large UML

Re: Broker side round robin on topic partitions when receiving messages

2020-06-15 Thread Colin McCabe
rtition balancing on the > broker - the producer sends messages directly to partition leaders so > partition currently needs to be defined on the producer. > > I understand that in order to perform round robin across partitions of a > topic when working with multiple producers, some develo

Re: [DISCUSS]: KIP-625: Richer encodings for integral-typed protocol fields.

2020-06-15 Thread Colin McCabe
Hi Tom, It's an interesting idea. Obviously protocol buffers does this for all numeric fields. I have to admit I have some mixed feelings, since this is another thing that makes encoding more complex. And it's not a clear win in all cases, although it is in some. I assume that the

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-06-12 Thread Colin McCabe
Hi Boyang, Thanks for the KIP! I think it's getting close. > For older requests that need redirection, forwarding > broker will just use its own authorizer to verify the principals. When the > request looks good, it will just forward the request with its own > credentials, no second

Re: Broker side round robin on topic partitions when receiving messages

2020-06-12 Thread Colin McCabe
HI Vinicius, This question seems like a better fit for the user mailing list rather than the developer mailing list. Anyway, if I understand correctly, you are asking if the producer can choose to assign partitions in a round-robin fashion rather than based on the key. The answer is, you

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-12 Thread Colin McCabe
gt; >> We added a new quota name in the KIP. You chose not to bump up the version > >> of DESCRIBE_CLIENT_QUOTAS and ALTER_CLIENT_QUOTAS, which seems ok since > >> the > >> quota name is represented as a string. However, the new quota name can be > >>

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-09 Thread Colin McCabe
lly be a little bit more specific, or at least tell the end user what to do with it (that's why I suggested "busy"). > > Voilà. I hope that I have addressed all your questions/points and I am > sorry for the lengthy email. > Thanks, David. It looks good to me overall. And I thought

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-08 Thread Colin McCabe
bout naming the error code BUSY? Then the error text could indicate the quota that we violated. This would be more generally useful as an error code and also avoid being confusingly similar to POLICY_VIOLATION. best, Colin > > I will let David chime in more on that. > > Thanks, >

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-07 Thread Colin McCabe
Hi David, Thanks for the KIP. I thought about this for a while and I actually think this approach is not quite right. The problem that I see here is that using an explicitly set quota here requires careful tuning by the cluster operator. Even worse, this tuning might be invalidated by

[jira] [Created] (KAFKA-10104) Remove deprecated --zookeeper flags as specified in KIP-604

2020-06-04 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-10104: Summary: Remove deprecated --zookeeper flags as specified in KIP-604 Key: KAFKA-10104 URL: https://issues.apache.org/jira/browse/KAFKA-10104 Project: Kafka

Re: [VOTE] KIP-601: Configurable socket connection timeout in NetworkClient

2020-06-03 Thread Colin McCabe
Hi Cheng, Thanks for working on this. Looks good. How about "socket.connection.setup.timeout.ms" and "socket.connection.setup.timeout.max.ms" (not connections with an S)? +1 (binding) best, Colin On Wed, Jun 3, 2020, at 06:24, Rajini Sivaram wrote: > Hi Cheng, > > Thanks for the updates,

[jira] [Resolved] (KAFKA-9945) TopicCommand should support --if-exists and --if-not-exists when --bootstrap-server is used

2020-06-02 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-9945. - Fix Version/s: 2.6.0 Reviewer: Colin McCabe Resolution: Fixed > TopicComm

Re: [VOTE] KIP-589: Add API to update Replica state in Controller

2020-05-27 Thread Colin McCabe
Thanks, David. +1 (binding). cheers, Colin On Wed, May 27, 2020, at 18:21, David Arthur wrote: > Colin, thanks for the feedback. Good points. I've updated the KIP with your > suggestions. > > -David > > On Wed, May 27, 2020 at 4:05 PM Colin McCabe wrote: > > &g

Re: [VOTE] KIP-589: Add API to update Replica state in Controller

2020-05-27 Thread Colin McCabe
Hi David, Thanks for the KIP! The KIP refers to "the KIP-500 bridge release (version 2.6.0 as of the time of this proposal)". This is out of date-- the bridge release will be one of the 3.x releases. We should either update this to 3.0, or perhaps just take out the reference to a specific

Re: [VOTE] KIP-578: Add configuration to limit number of partitions

2020-05-27 Thread Colin McCabe
Hi Gokul Ramanan Subramanian, Thanks for the KIP. Can you please modify the KIP to remove the reference to the deprecated --zookeeper flag? This is not how kafka-configs.sh is supposed to be used in new versions of Kafka. You get a warning message if you do use this deprecated flag. As

[jira] [Resolved] (KAFKA-9942) ConfigCommand fails to set client quotas for default users with --bootstrap-server.

2020-05-21 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-9942. - Fix Version/s: 2.5.1 2.6.0 Resolution: Fixed > ConfigCommand fa

[jira] [Resolved] (KAFKA-9980) Fix bug where alterClientQuotas could not set default client quotas

2020-05-21 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-9980. - Fix Version/s: 2.5.1 Resolution: Fixed > Fix bug where alterClientQuotas could not

Re: [VOTE]: KIP-604: Remove ZooKeeper Flags from the Administrative Tools

2020-05-21 Thread Colin McCabe
0 at 5:41 PM Colin McCabe wrote: > > > On Tue, May 19, 2020, at 09:31, Jason Gustafson wrote: > > > Hi Colin, > > > > > > Looks good. I just had one question. It sounds like your intent is to > > > change kafka-configs.sh so that the --zookeeper flag i

Re: [VOTE]: KIP-604: Remove ZooKeeper Flags from the Administrative Tools

2020-05-19 Thread Colin McCabe
t; > > > > +1 (non-binding). > > > > > > > > Thanks for the KIP. > > > > > > > > On Fri, May 15, 2020 at 12:41 AM Guozhang Wang > > wrote: > > > > > > > > > +1. > > > > > &g

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

2020-05-19 Thread Colin McCabe
ng to connect to a single fixed node, like the controller node) best, Colin > > Regards, > > Rajini > > > On Mon, May 18, 2020 at 8:48 PM Colin McCabe wrote: > > > Hi Rajini, > > > > I think the idea behind the 10 second default is that if you

Re: [VOTE] KIP-601: Configurable socket connection timeout in NetworkClient

2020-05-18 Thread Colin McCabe
Java NIO doesn't dictate how long we should wait before terminating the attempt to connect. Hope I didn't miss anything. best, Colin > > Best, - Cheng Tan > > > > > On May 18, 2020, at 1:32 PM, Colin McCabe wrote: > > > > Hi Cheng, > > > > socket.connectio

[jira] [Resolved] (KAFKA-9292) KIP-551: Expose disk read and write metrics

2020-05-18 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-9292. - Fix Version/s: 2.6.0 Resolution: Fixed > KIP-551: Expose disk read and write metr

Re: [VOTE] KIP-601: Configurable socket connection timeout in NetworkClient

2020-05-18 Thread Colin McCabe
Hi Cheng, socket.connection.setup.timeout.ms seems more consistent with our existing configuration names than socket.connections.setup.timeout.ms (with an s). What do you think? > If no connected or connecting node exists, provide the disconnected node which > respects the reconnect backoff

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

2020-05-18 Thread Colin McCabe
Hi Rajini, I think the idea behind the 10 second default is that if you have three Kafka nodes A, B, C (or whatever), and you can't talk to A within 10 seconds, you'll try again with B or C, and still have plenty of time left over. Whereas currently, if your connection hangs while trying to

Re: [DISCUSS] KIP-573: Enable TLSv1.3 by default

2020-05-18 Thread Colin McCabe
Hmm. It would be good to figure out if we are going to remove this compatibility hack in the next major release of Kafka? In other words, in Kafka 3.0, will we enable TLS 1.3 by default even if the cipher suite is specified? best, Colin On Mon, May 18, 2020, at 09:26, Ismael Juma wrote: >

Re: [VOTE] KIP-608: Expose Kafka Metrics in Authorizer

2020-05-18 Thread Colin McCabe
Hi Zhiguo, I think it's really important to distinguish between "a class being public" and "a class being part of Kafka's public API." These really are two completely different concepts, unfortunately. The best explanation of interface annotations is probably here:

Re: [VOTE] KIP-608: Expose Kafka Metrics in Authorizer

2020-05-18 Thread Colin McCabe
KafkaMetrics isn't a public API currently. Are we willing to make it one? I think this is a very big change, if so. This affects a huge number of classes. MetricConfig, MetricReporter, MetricName, Sensor, KafkaMetric, and probably more I'm forgetting would need to become public APIs that we

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-14 Thread Colin McCabe
803y67WSXMYkgh77k=,iterations=4096] > > > Please let me know what you think. > > Best, - Cheng Tan > > > On Apr 30, 2020, at 11:16 PM, Colin McCabe wrote: > > > > > >

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-14 Thread Colin McCabe
NqamVpeml6NGhiZmMwY3hrbg==,stored_key=sb5jkqStV9RwPVTGxG1ZJHxF89bqjsD1jT4SFDK4An2goSnWpbNdY0nkq0fNV8xFcZqb7MVMJ1tyEgif5OXKDQ==, > > server_key=3EfuHB4LPOcjDH0O5AysSSPiLskQfM5K9+mOzGmkixasmWEGJWZv7svtgkP+acO2Q9ms9WQQ9EndAJCvKHmjjg==,iterations=4096],SCRAM-SHA-256=[salt=10ibs0z7xzlu6w5ns0n188sis5,s

  1   2   3   4   5   6   7   8   9   10   >