Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-06 Thread Colin McCabe
ds, etc.) best, Colin > > Kind regards, > > Tom > > Cheers, > > Tom > > On Tue, May 5, 2020 at 12:52 AM Guozhang Wang wrote: > > > Cool, that makes sense. > > > > Guozhang > > > > > > On Mon, May 4, 2020 at 2:50 PM Colin

Re: [DISCUSS] Kafka 3.0

2020-05-06 Thread Colin McCabe
ted to removing Zookeeper could land in pre-3.0 while every other > > > > > feature targets 2.6. > > > > > > > > > > If you are proposing 2.6 should be the "bridge release", I think this > > > is > > > > > premature given

Re: [DISCUSS] Kafka 3.0

2020-05-06 Thread Colin McCabe
legacy MirrorMaker before 3.0 as well. > > I'm happy to drive that additional breaking change. Maybe 2.6 can be the > > "bridge" for MM2 as well. > > > > Ryanne > > > > On Mon, May 4, 2020, 5:05 PM Colin McCabe wrote: > > > > > Hi all,

Re: [DISCUSS] Kafka 3.0

2020-05-06 Thread Colin McCabe
as well. I don't have a strong opinion either way about this, but if we want to remove the original MirrorMaker, we have to deprecate it first, right? Are we ready to do that? best, Colin > > Ryanne > > On Mon, May 4, 2020, 5:05 PM Colin McCabe wrote: > > > Hi all, >

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

2020-05-04 Thread Colin McCabe
l clients, not just AdminClient? best, Colin On Mon, May 4, 2020, at 13:18, Colin McCabe wrote: > Hmm. A big part of the reason behind the KIP is that the default > connection timeout behavior of the OS doesn't work for Kafka, right? > For example, on Linux, if we wait 127 seconds

[DISCUSS] Kafka 3.0

2020-05-04 Thread Colin McCabe
Hi all, We've had a few proposals recently for incompatible changes. One of them is my KIP-604: Remove ZooKeeper Flags from the Administrative Tools. The other is Boyang's KIP-590: Redirect ZK Mutation Protocols to the Controller. I think it's time to start thinking about Kafka 3.0.

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-04 Thread Colin McCabe
ow that quotas may be specified in the former pattern as well. If we > believe that such hierarchical configuration may be more common in the > future, I'm wondering should we just consider support it more natively in > alter/describe config patterns. > > > Guozhang > >

Re: [DISCUSS] KIP-604: Remove ZooKeeper Flags from the Administrative Tools

2020-05-04 Thread Colin McCabe
1] > https://cwiki.apache.org/confluence/display/KAFKA/KIP-460%3A+Admin+Leader+Election+RPC > > On Fri, 1 May 2020 at 08:17, Colin McCabe wrote: > > > Hi all, > > > > I posted a KIP about removing the deprecated --zookeeper flags from our > > administrative tool

Re: [DISCUSS] KIP-589 Add API to Update Replica State in Controller

2020-05-04 Thread Colin McCabe
> > > In the rejecting the alternative of having an RPC for log dir failures > > you > > > say: > > > > > > It was also rejected to prevent "leaking" the notion of a log dir to the > > > > public API. > > > > > > > &

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-04 Thread Colin McCabe
ation, > as well as error conditions? > > Guozhang > > > On Fri, May 1, 2020 at 5:12 PM Colin McCabe wrote: > > > On Fri, May 1, 2020, at 08:35, Aneel Nazareth wrote: > > > Hi Colin, > > > > > > Thanks for the KIP. Is it also in scope to

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

2020-05-04 Thread Colin McCabe
Hmm. A big part of the reason behind the KIP is that the default connection timeout behavior of the OS doesn't work for Kafka, right? For example, on Linux, if we wait 127 seconds for a connection attempt to time out, we won't get a chance to make another attempt in most cases. So I think it

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-01 Thread Colin McCabe
ibed in the KIP. best, Colin > On Fri, May 1, 2020 at 1:18 AM Colin McCabe wrote: > > > > Hi all, > > > > I posted a KIP about adding a new SCRAM configuration API on the broker. > > Check it out here if you get a chance: > > https://cwiki.apache.org/confluence/x/ihERCQ > > > > cheers, > > Colin >

[DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-01 Thread Colin McCabe
Hi all, I posted a KIP about adding a new SCRAM configuration API on the broker. Check it out here if you get a chance: https://cwiki.apache.org/confluence/x/ihERCQ cheers, Colin

[DISCUSS] KIP-604: Remove ZooKeeper Flags from the Administrative Tools

2020-05-01 Thread Colin McCabe
Hi all, I posted a KIP about removing the deprecated --zookeeper flags from our administrative tools. Check it out here: https://cwiki.apache.org/confluence/x/kRARCQ best, Colin

[jira] [Created] (KAFKA-9945) TopicCommand should support --if-exists and --if-not-exists when --bootstrap-server is used

2020-04-30 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-9945: --- Summary: TopicCommand should support --if-exists and --if-not-exists when --bootstrap-server is used Key: KAFKA-9945 URL: https://issues.apache.org/jira/browse/KAFKA-9945

[jira] [Resolved] (KAFKA-7740) Kafka Admin Client should be able to manage user/client configurations for users and clients

2020-04-29 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-7740. - Resolution: Fixed > Kafka Admin Client should be able to manage user/client configurati

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

2020-04-28 Thread Colin McCabe
Hi Jason, It's amazing to see this coming together :) I haven't had a chance to read in detail, but I read the outline and a few things jumped out at me. First, for every epoch that is 32 bits rather than 64, I sort of wonder if that's a good long-term choice. I keep reading about stuff like

Re: [DISCUSS] KIP-589 Add API to Update Replica State in Controller

2020-04-28 Thread Colin McCabe
Hi David, Thanks for the KIP! I think the ReplicaStateEventResponse should have a separate error code for each partition. Currently it just has one error code for the whole request/response, if I'm reading this right. I think Jose made a similar point as well. We should plan for scenarios

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

2020-04-28 Thread Colin McCabe
Hi Cheng, Thanks for the KIP. > Currently, the initial socket connection timeout is depending on system > setting tcp_syn_retries. The actual timeout value is 2 ^ (tcp_sync_retries + > 1) - 1 seconds This section is confusing since it refers to Linux configuration settings without mentioning

Re: [VOTE] KIP-584: Versioning scheme for features

2020-04-27 Thread Colin McCabe
y/KAFKA/KIP-584%3A+Versioning+scheme+for+features#KIP-584:Versioningschemeforfeatures-Controller:ZKnodebootstrapwithdefaultvalues > > > Cheers, > Kowshik > > On Fri, Apr 24, 2020 at 8:50 PM Colin McCabe wrote: > > > On Fri, Apr 24, 2020, at 00:01, Kowshik Prakasam

Re: [VOTE] KIP-584: Versioning scheme for features

2020-04-24 Thread Colin McCabe
on_level" becomes useful. Using this, a client could learn the > specific supported versions that's lower than max_version_level (instead of > assuming that all brokers support the range: [1, max_version_level]). For > example, if the cluster deprecates "min_version_level", then t

Re: [VOTE] KIP-584: Versioning scheme for features

2020-04-23 Thread Colin McCabe
Hi Kowshik, Thanks again for working on this-- it looks great. I went over the KIP again and have a few more comments. === It would be good to note that deprecating a feature version (in other words, increasing minVersionLevel on the broker) is an incompatible change, which requires a major

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-23 Thread Colin McCabe
> > > > Guozhang > > > > > > On Wed, Apr 22, 2020 at 1:47 PM Boyang Chen > > > wrote: > > > > > > > Thanks Colin for the summary! And Guozhang, regarding the future use > > > cases, > > > > consider a scenario where there are temporar

[jira] [Created] (KAFKA-9905) The equals functions for generated classes should compare all fields

2020-04-22 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-9905: --- Summary: The equals functions for generated classes should compare all fields Key: KAFKA-9905 URL: https://issues.apache.org/jira/browse/KAFKA-9905 Project: Kafka

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-22 Thread Colin McCabe
Hi all, I guess the way I see this working is that the request gets sent from the client, to the initial broker, and then forwarded to the final broker. The initial broker should do authentication (who are you?) and come up with a principal name. Then it creates an envelope request, which

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-18 Thread Colin McCabe
x just by using the ZK version, but there are other race conditions like what if we're deleting a topic while setting this config, etc. A single writer is a lot easier to reason about. best, Colin > > On Fri, Apr 17, 2020 at 12:53 PM Colin McCabe wrote: > > > On Thu, Apr 16, 202

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-17 Thread Colin McCabe
On Thu, Apr 16, 2020, at 12:33, Jose Garcia Sancio wrote: > Hi Boyang, > > Thanks for the KIP. The KIP looks good. I have a few questions and comments. > > > As part of the KIP-500 >

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-17 Thread Colin McCabe
; Ismael > > On Wed, Apr 15, 2020, 5:10 PM Colin McCabe wrote: > > > Hi Ismael, > > > > I agree that sending these requests through the controller will not work > > during the periods when there is no controller. However, those periods > > should be s

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-17 Thread Colin McCabe
th the rational here. The new `Envelope` RPC has been added to the KIP > > and the forwarding section logic has been revised, feel free to take > > another look. > > > > On Wed, Apr 15, 2020 at 5:19 PM Colin McCabe wrote: > > > > > Hi Boyang, > > >

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-15 Thread Colin McCabe
> > some purgatory to keep the forwarded requests; if that's true, > > should > > > > we > > > > > > add some broker-side metrics for those purgatories for debugging > > > > > purposes? > > > > > > > > > > > > 2) Should we also consider adding so

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-15 Thread Colin McCabe
Hi Ismael, I agree that sending these requests through the controller will not work during the periods when there is no controller. However, those periods should be short-- otherwise we have bigger problems in the cluster. These requests are very infrequent because they are administrative

Re: [DISCUSS] KIP-587 Suppress detailed responses for handled exceptions in security-sensitive environments

2020-04-15 Thread Colin McCabe
> > > Our particular need was around suppressing exceptions in the > "public" > > response, as Kafka Connect was passing these exceptions without > > authentication, they became a public endpoint upon which the > auditors coul

Re: [kafka-clients] [VOTE] 2.5.0 RC3

2020-04-13 Thread Colin McCabe
+1 (binding) verified checksums ran unitTest ran check best, Colin On Tue, Apr 7, 2020, at 21:03, David Arthur wrote: > Hello Kafka users, developers and client-developers, > > This is the forth candidate for release of Apache Kafka 2.5.0. > > * TLS 1.3 support (1.2 is now the default) > *

Re: [DISCUSS] (KAFKA-9806) authorize cluster operation when creating internal topics

2020-04-13 Thread Colin McCabe
the internal topic. > Let me know if you have any feedback. Hi Paolo, If the problem is broker ACLs being configured incorrectly so that it can't receive requests from the controller, a lot of things will fail. This isn't really related to anything with FindCoordinator. best, Colin >

Re: [DISCUSS] (KAFKA-9806) authorize cluster operation when creating internal topics

2020-04-07 Thread Colin McCabe
r FindCoordinatorRequest. best, Colin > > Thanks, > Paolo > > On Mon, 6 Apr 2020 at 23:58, Colin McCabe wrote: > > > Hi Paolo, > > > > Thanks for finding this issue. > > > > Unfortunately, you certainly can't add a new permission requirement

Re: [DISCUSS] (KAFKA-9806) authorize cluster operation when creating internal topics

2020-04-06 Thread Colin McCabe
Hi Paolo, Thanks for finding this issue. Unfortunately, you certainly can't add a new permission requirement to an existing RPC without breaking compatibility. So the current solution in the PR will not work. However, you should be able to have the broker create the topic using its own

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-06 Thread Colin McCabe
Hi Sönke, Yeah, that was my thought too. The request has already been validated on the forwarding broker, so we don't need to validate it again. However, you make a good point that it's unfortunate that the audit log would lose the principal information if we didn't forward it as well.

Re: [DISCUSS] KIP-584: Versioning scheme for features

2020-04-06 Thread Colin McCabe
ives UpdateMetadataRequest from controller. Anything the > > > broker > > > > is > > > > > > doing up > > > > > > until this point is not damaging/useful. > > > > > > > > > > > > I’ve clarified the above in

[jira] [Created] (KAFKA-9825) Kafka protocol BNF format should have some way to display tagged fields

2020-04-06 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-9825: --- Summary: Kafka protocol BNF format should have some way to display tagged fields Key: KAFKA-9825 URL: https://issues.apache.org/jira/browse/KAFKA-9825 Project: Kafka

Re: [DISCUSS] KIP-584: Versioning scheme for features

2020-04-04 Thread Colin McCabe
that user could configure a range of > > > > > supported versions, what's the trade-off for allowing single > > finalized > > > > > version only? > > > > > > > > (Kowshik): Great question! The finalized version of a feature basically > > >

Re: [DISCUSS] KIP-584: Versioning scheme for features

2020-04-04 Thread Colin McCabe
rsistence, not mutating critical in-memory state, won’t be > >> > > > serving > >> > > > produce/fetch requests. Note it doesn’t even know it’s assigned > >> > > partitions > >> > > > until > >> > > > it receive

[jira] [Created] (KAFKA-9817) Poor performance for electLeaders controller RPC

2020-04-03 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-9817: --- Summary: Poor performance for electLeaders controller RPC Key: KAFKA-9817 URL: https://issues.apache.org/jira/browse/KAFKA-9817 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-587 Suppress detailed responses for handled exceptions in security-sensitive environments

2020-04-03 Thread Colin McCabe
Also, if you do find a security issue, the process to follow is here: https://kafka.apache.org/project-security.html . best, Colin On Fri, Apr 3, 2020, at 14:20, Colin McCabe wrote: > Hi Connor, > > If we are putting security-sensitive information into REST responses, > t

Re: [DISCUSS] KIP-587 Suppress detailed responses for handled exceptions in security-sensitive environments

2020-04-03 Thread Colin McCabe
Hi Connor, If we are putting security-sensitive information into REST responses, that is a bug that needs to be fixed, not worked around with a configuration option. Do you have an example of security-sensitive information appearing in the exception text? Why do you feel that PCI-DSS

Re: [DISCUSS] KIP-584: Versioning scheme for features

2020-04-03 Thread Colin McCabe
heme+for+features#KIP-584:Versioningschemeforfeatures-Incompatiblebrokerlifetime > > > >> > . > > > >> > > > > >> > > 5. "adding a new Feature, updating or deleting an existing > > Feature", > > > >> may >

Re: [VOTE] KIP-574: CLI Dynamic Configuration with file input

2020-03-30 Thread Colin McCabe
ing) > > > > > > > > Thanks for the KIP, Aneel! > > > > > > > > Regards, > > > > > > > > Rajini > > > > > > > > On Wed, Mar 25, 2020 at 5:06 AM Gwen Shapira > > wrote: > > > > > &

Re: [DISCUSS] KIP-574: CLI Dynamic Configuration with file input

2020-03-30 Thread Colin McCabe
n command line, > > > they can always use the existing --add-config option. > > > > > > > > > > > > > > > On Thu, Mar 26, 2020 at 7:20 PM David Jacot wrote: > > > > > > > Rajini has made a good point. I don't feel strong for eit

Re: [DISCUSS] KIP-584: Versioning scheme for features

2020-03-30 Thread Colin McCabe
eCommand tool - you are right! I've updated > the KIP sketching the functionality provided by this tool, with some > examples. Please look at the section "Tooling support examples". > > Thank you! Thanks, Kowshik. cheers, Colin > > > Cheers, > Kowshik > &

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

2020-03-30 Thread Colin McCabe
r pattern when producer throws some > > processing may already been done with irreversible side-effects) so that > > users are aware. > > > > > > Guozhang > > > > > > On Sun, Mar 29, 2020 at 9:07 PM Colin McCabe wrote: > > > > &

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

2020-03-29 Thread Colin McCabe
ent version. On code level, the path to call transaction > >> commit > >>> is better to be unified instead of diverged with try-catch. As a matter > >> of > >>> fact, we have already implemented the downgrade of the OffsetFetch > >> protocol > >>>

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

2020-03-27 Thread Colin McCabe
On Fri, Mar 27, 2020, at 18:29, Colin McCabe wrote: > On Fri, Mar 27, 2020, at 16:06, Boyang Chen wrote: > > Hey there, > > > > we would like to address an improvement on the > > *Producer#sendOffsetsToTransaction(offsets, > > groupMetadata) *API. Previously we as

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

2020-03-27 Thread Colin McCabe
On Fri, Mar 27, 2020, at 16:06, Boyang Chen wrote: > Hey there, > > we would like to address an improvement on the > *Producer#sendOffsetsToTransaction(offsets, > groupMetadata) *API. Previously we assume the calling of this function > would crash the app if broker is not on version 2.5.0 or

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2020-03-26 Thread Colin McCabe
. > > Thanks again to everybody who helped discuss, vote on, review and implement > this KIP. :) > > On Wed, Aug 28, 2019 at 8:47 PM Colin McCabe wrote: > > > Hi all, > > > > After some discussion with Jun and Stan, we decided that we should bump > > the versio

Re: [DISCUSS] KIP-584: Versioning scheme for features

2020-03-26 Thread Colin McCabe
Thanks, Kowshik, this looks good. In the "Schema" section, do we really need both __schema_version__ and __data_version__? Can we just have a single version field here? Shouldn't the Admin(Client) function have some way to get the min and max information that we're exposing as well? I guess

Re: [DISCUSS] KIP-574: CLI Dynamic Configuration with file input

2020-03-26 Thread Colin McCabe
gt; though the config that is in the file is a=1? Or would that be > > > > expected? > > > > > > > > On Mon, Mar 9, 2020 at 1:02 PM David Jacot > > wrote: > > > > > > > > > > Hi Colin, > > > > > > > >

Re: [DISCUSS] KIP-574: CLI Dynamic Configuration with file input

2020-03-26 Thread Colin McCabe
> But what if this were run in-between: > > > > > > > > > > bin/kafka-configs --bootstrap-server localhost:9092 \ > > > > > --entity-type brokers --entity-default \ > > > > > --alter --add-config a=3 &

[jira] [Created] (KAFKA-9761) kafka-consumer-groups tool overrides admin client defaults with a short 5 s timeout

2020-03-25 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-9761: --- Summary: kafka-consumer-groups tool overrides admin client defaults with a short 5 s timeout Key: KAFKA-9761 URL: https://issues.apache.org/jira/browse/KAFKA-9761

Re: [VOTE] KIP-574: CLI Dynamic Configuration with file input

2020-03-18 Thread Colin McCabe
+1 (binding). Thanks, Aneel. best, Colin On Wed, Mar 18, 2020, at 00:04, David Jacot wrote: > +1 (non-binding), thanks for the KIP! > > David > > On Mon, Mar 16, 2020 at 4:06 PM Aneel Nazareth wrote: > > > Hello all, > > > > Thanks to the folks who have given feedback. I've incorporated the

Re: [VOTE] KIP-518: Allow listing consumer groups per state

2020-03-12 Thread Colin McCabe
t; dja...@confluent.io > wrote: > > > > > > > > > > > > > > +1 (non-binding). Thanks for the KIP! > > > > > > > > > > > > David > > > > > > > > > > > > On Thu, Feb 6, 2020 at

[jira] [Resolved] (KAFKA-9364) Fix misleading consumer logs on throttling

2020-03-12 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-9364. - Fix Version/s: 2.5.0 Resolution: Fixed > Fix misleading consumer logs on throttl

[jira] [Created] (KAFKA-9688) kafka-topic.sh should show KIP-455 adding and removing replicas

2020-03-09 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-9688: --- Summary: kafka-topic.sh should show KIP-455 adding and removing replicas Key: KAFKA-9688 URL: https://issues.apache.org/jira/browse/KAFKA-9688 Project: Kafka

Re: [DISCUSS] KIP-574: CLI Dynamic Configuration with file input

2020-03-09 Thread Colin McCabe
Hi Aneel, Thanks for the KIP. I like the idea. You mention that "input from STDIN can be used instead of a file on disk." The example given in the KIP seems to suggest that the command defaults to reading from STDIN if no argument is given to --add-config-file. I would argue against this

[jira] [Created] (KAFKA-9687) Improve Trogdor workload exit logging

2020-03-09 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-9687: --- Summary: Improve Trogdor workload exit logging Key: KAFKA-9687 URL: https://issues.apache.org/jira/browse/KAFKA-9687 Project: Kafka Issue Type: Improvement

[jira] [Resolved] (KAFKA-5722) Refactor ConfigCommand to use the AdminClient

2020-03-06 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-5722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-5722. - Resolution: Duplicate > Refactor ConfigCommand to use the AdminCli

Re: Create content permission

2020-03-06 Thread Colin McCabe
Added best, Colin On Fri, Mar 6, 2020, at 09:45, Cheng Tan wrote: > It’s d8tltanc. Thank you, > > > On Mar 6, 2020, at 9:35 AM, Matthias J. Sax wrote: > > > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA512 > > > > What is your wiki account id? > > > > - -Matthias > > > > On 3/5/20

[jira] [Resolved] (KAFKA-9635) Should ConfigProvider.subscribe be deprecated?

2020-03-06 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-9635. - Resolution: Duplicate Let's continue the discussion on the mailing list. > Sho

Re: Subject: [VOTE] 2.4.1 RC0

2020-03-06 Thread Colin McCabe
+1 (binding) Checked the git hash and branch, looked at the docs a bit. Ran quickstart (although not the connect or streams parts). Looks good. best, Colin On Fri, Mar 6, 2020, at 07:31, David Arthur wrote: > +1 (binding) > > Download kafka_2.13-2.4.1 and verified signature, ran

[jira] [Resolved] (KAFKA-8345) Create an Administrative API for Replica Reassignment

2020-03-02 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-8345. - Fix Version/s: 2.4.0 Resolution: Fixed > Create an Administrative API for Repl

Re: [DISCUSS] KIP-569: DescribeConfigsResponse - Update the schema to include datatype of the field

2020-02-28 Thread Colin McCabe
On Fri, Feb 28, 2020, at 14:00, Shailesh Panwar wrote: > On Fri, Feb 28, 2020 at 11:04 AM Colin McCabe wrote: > > > On Thu, Feb 27, 2020, at 11:37, Shailesh Panwar wrote: > > > Thanks Colin for the inputs. Regarding > > > > > > > I think most users of

[jira] [Created] (KAFKA-9625) Broker throttles are incorrectly marked as sensitive configurations

2020-02-28 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-9625: --- Summary: Broker throttles are incorrectly marked as sensitive configurations Key: KAFKA-9625 URL: https://issues.apache.org/jira/browse/KAFKA-9625 Project: Kafka

Re: [DISCUSS] KIP-569: DescribeConfigsResponse - Update the schema to include datatype of the field

2020-02-28 Thread Colin McCabe
s that there will be some version X client that doesn't understand all the possible configuration types that a version X + 1 client and broker understand. What does the version X client see in this case when it uses the API? best, Colin > > Thanks > Shailesh > > On 2020/02/26 2

Re: [DISCUSS] KIP-569: DescribeConfigsResponse - Update the schema to include datatype of the field

2020-02-26 Thread Colin McCabe
Hi Shailesh, I think most users of the DescribeConfigs API do not want to get help text or configuration schema information. So, it would be inefficient to always include this information as part of the DescribeConfigs response. It would be better to create a new, separate API for getting

Re: [DISCUSS] KIP-518: Allow listing consumer groups per state

2020-02-19 Thread Colin McCabe
ibe" and we will have "--states" which can be used with > > "--list". I feel this > > is going to be confusing for users. I wonder if we could combine both cases > > to reduce > > the confusion but I am not sure it would be better. What do you think? &

Re: [VOTE] KIP-518: Allow listing consumer groups per state

2020-02-06 Thread Colin McCabe
Hi Mickael, Thanks for the KIP. I left a comment on the DISCUSS thread as well. best, Colin On Thu, Feb 6, 2020, at 08:58, Mickael Maison wrote: > Hi Manikumar, > > I believe I've answered David's comments in the DISCUSS thread. > Thanks > > On Wed, Jan 15, 2020 at 10:15 AM Manikumar

Re: [DISCUSS] KIP-518: Allow listing consumer groups per state

2020-02-06 Thread Colin McCabe
Hi Mickael, Can you please specify what the result is when a newer client tries to use this on an older broker? Does that result in an UnsupportedVersionException? I would prefer an Optional in the Java API so that “show all groups” can be EMPTY. best, Colin On Mon, Jan 27, 2020, at 07:53,

[jira] [Created] (KAFKA-9506) Fix JmxReporter corner cases and optimize

2020-02-05 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-9506: --- Summary: Fix JmxReporter corner cases and optimize Key: KAFKA-9506 URL: https://issues.apache.org/jira/browse/KAFKA-9506 Project: Kafka Issue Type

Re: [DISCUSS] 2.4.1 Bug Fix Release

2020-02-04 Thread Colin McCabe
Hi Bill, Sounds good. +1. best, Colin On Mon, Feb 3, 2020, at 17:27, Bill Bejeck wrote: > Hi All, > > I'd like to volunteer for the release manager of the 2.4.1 bug fix release. > > Kafka 2.4.0 was released on December 16, 2019, and so far 19 issues have > been fixed since then. > > Here is

Re: [VOTE] KIP-546: Add quota-specific APIs to the Admin Client, redux

2020-01-27 Thread Colin McCabe
Thanks, Brian. +1 (binding) best, Colin On Sun, Jan 26, 2020, at 07:56, Brian Byrne wrote: > Hello all, > > I'd like to start a vote on KIP-456: Add quota-specific APIs to the Admin > Client, redux > > The KIP is here: >

Re: [DISCUSS] KIP-546: Add quota-specific APIs to the Admin Client, redux

2020-01-25 Thread Colin McCabe
hare his thinking when he's back. The > > > > > > > QuotaFilter > > > > > > > > > usage was an error, this has been corrected. > > > > > > > > > > > > > > > > > > For (2), the config-centric mode is w

Re: [VOTE] KIP-555: An admin tools proposal to accomplish the deprecation of zookeeper access that is direct

2020-01-17 Thread Colin McCabe
ing) > > Thanks for the KIP! > > David > > Le ven. 17 janv. 2020 à 19:25, Colin McCabe a écrit : > > > Hi all, > > > > I'm going to close the vote later today. > > > > thanks, > > Colin > > > > > > On Wed, Jan 15, 2020, at 06

Re: [VOTE] KIP-555: An admin tools proposal to accomplish the deprecation of zookeeper access that is direct

2020-01-17 Thread Colin McCabe
d, Jan 15, 2020 at 2:48 AM Gwen Shapira wrote: > > > > > > > +1 (binding, re-vote) > > > > > > > > On Tue, Jan 14, 2020 at 11:23 AM Colin McCabe > > > wrote: > > > > > > > > > > Hi all, > > >

Re: [DISCUSS] KIP-546: Add quota-specific APIs to the Admin Client, redux

2020-01-17 Thread Colin McCabe
vant but might be in the future if a different quota gets deleted. One way to do this would be to have two maps-- one for applicable quotas and one for shadowed quotas. best, Colin On Tue, Jan 14, 2020, at 13:32, Brian Byrne wrote: > Hi Colin, > > Your feedback is appreciated, thank

Re: [VOTE] KIP-551: Expose disk read and write metrics

2020-01-16 Thread Colin McCabe
With binding +1 votes from Gwen Shapira, Manikumar Reddy, Mickael Maison, and David Arthur, and non-binding +1 votes from Jose Garcia Sancio, M. Manna, Lucas Bradstreet, Mitchell, and Sönke Liebau, the vote passes. Thanks, all! Colin On Wed, Jan 15, 2020, at 09:25, Colin McCabe wrote

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

2020-01-16 Thread Colin McCabe
if user configures properties > explicitly. In the current approach, this was simpler since we just had to > set the configured values. But if we decide to inherit Kafka configs, then > we will need to check system properties and update accordingly. But that > should be doable? >

Re: [DISCUSS] KIP-550: Mechanism to Delete Stray Partitions on Broker

2020-01-16 Thread Colin McCabe
rapid deletion. These bugs didn't manifest in testing or routine upgrades. best, Colin > > Thanks, > Dhruvil > > On Thu, Jan 16, 2020 at 3:24 AM Colin McCabe wrote: > > > On Wed, Jan 15, 2020, at 03:54, Dhruvil Shah wrote: > > > Hi Colin, > > > > >

Re: [DISCUSS] KIP-550: Mechanism to Delete Stray Partitions on Broker

2020-01-15 Thread Colin McCabe
rators to what is about to happen. best, Colin > > Thanks, > Dhruvil > > On Tue, Jan 14, 2020 at 11:04 PM Colin McCabe wrote: > > > Hi Dhruvil, > > > > Thanks for the KIP. I think there should be some way to turn this off, in > > case that becomes ne

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

2020-01-15 Thread Colin McCabe
; system property. (I said it was an idea -- not necessarily a good one > :-) > > Ron > > On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe wrote: > > > > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote: > > > Thanks Colin and Rajini. > > > > > >

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

2020-01-15 Thread Colin McCabe
get us there. > > Ron I think, on balance, I agree with Rajini that it would be nice to make the configs look more like Kafka configs. We don't have camel-case in configuration keys, so we should avoid it here. best, Colin > > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe wrote: &

Re: [VOTE] KIP-551: Expose disk read and write metrics

2020-01-15 Thread Colin McCabe
seful for folks managing a cluster by themselves. > > > > > > > > > > > > > > M. MAnna > > > > > > > > > > > > > > On Fri, 10 Jan 2020 at 22:35, Jose Garcia Sancio < > >

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

2020-01-15 Thread Colin McCabe
but the > > special > > > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials > > > prior to starting Kafka will still exist, so it feels like while > > > "--zk-tls-config-file" would never be used except for this use case it >

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

2020-01-15 Thread Colin McCabe
>> > > >>>>> On Tue, Jan 14, 2020 at 6:32 PM Mickael Maison < > > >>> mickael.mai...@gmail.com> > > >>>>> wrote: > > >>>>> > > >>>>>> Congrats Colin, Vahid and Manikumar! > > >>>

Re: [DISCUSS] KIP-546: Add quota-specific APIs to the Admin Client, redux

2020-01-14 Thread Colin McCabe
Hi Brian, Thanks for the KIP. The KIP says: > As represented by the current ZK node structure, the order in which quotas > are matched are as follows. Note is a specified user principal, > is a specified client ID, and is a special default > user/client ID that matches to all users or

[VOTE] KIP-555: An admin tools proposal to accomplish the deprecation of zookeeper access that is direct

2020-01-14 Thread Colin McCabe
Hi all, I'm reposting this since I've been informed that gmail mashed the original VOTE thread into a different email thread. Hopefully the different thread title will prevent it doing that in this case. I'd like to start the vote on KIP-555: Deprecate Direct Zookeeper access in Kafka

[VOTE] KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools

2020-01-14 Thread Colin McCabe
Hi all, I'd like to start the vote on KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools. KIP: https://cwiki.apache.org/confluence/x/Wg6dC Discussion thread:

Re: [DISCUSS] KIP-550: Mechanism to Delete Stray Partitions on Broker

2020-01-14 Thread Colin McCabe
Hi Dhruvil, Thanks for the KIP. I think there should be some way to turn this off, in case that becomes necessary. I'm also curious how long we intend to wait between detecting the duplication and deleting the extra logs. The KIP says "scheduled for deletion" but doesn't give a time frame

Re: [DISCUSS] KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools

2020-01-14 Thread Colin McCabe
>> wrote: > >> > >> Comments below. > >> > >>> On Thu, Jan 9, 2020 at 5:03 PM Colin McCabe wrote: > >>> > >>> That's a good question. The current plan is for the 3.x releases to still > >>> require ZooKeeper. What

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

2020-01-14 Thread Colin McCabe
-config-file" would never be used except for this use case it > could probably still be added for this particular situation. > > Ron > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I > sent) did not go through. I am trying to get emails throug

Re: [DISCUSS] Apache Kafka 2.5.0 release

2020-01-13 Thread Colin McCabe
+1. Thanks for volunteering, David. best, Colin On Fri, Dec 20, 2019, at 10:59, David Arthur wrote: > Greetings! > > I'd like to volunteer to be release manager for the next time-based feature > release which will be 2.5. If there are no objections, I'll send out the > release plan in the next

[VOTE] KIP-551: Expose disk read and write metrics

2020-01-10 Thread Colin McCabe
Hi all, I'd like to start the vote on KIP-551: Expose disk read and write metrics. KIP: https://cwiki.apache.org/confluence/x/sotSC Discussion thread: https://lists.apache.org/thread.html/cfaac4426455406abe890464a7f4ae23a5c69a39afde66fe6eb3d696%40%3Cdev.kafka.apache.org%3E cheers, Colin

Re: [DISCUSS] KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools

2020-01-10 Thread Colin McCabe
e tools that only support zookeeper (zookeeper-security-migration.sh > > and zookeeper-shell.sh) should we be deprecating the entire tool for > > removal in a future 3.0 release? > > > > On Thu, Jan 9, 2020 at 4:24 PM Colin McCabe wrote: > > > > > Hi all, > >

<    3   4   5   6   7   8   9   10   11   12   >