Re: Why should SASL principal be unchanged upon reauth

2020-09-28 Thread Gokul Ramanan Subramanian
iscussion. I’m not aware of anything specific that would prevent the > change, but I will say that my gut instinct says it wouldn’t be a good idea > without a truly compelling justification. > > Ron > > > On Sep 28, 2020, at 8:34 AM, Gokul Ramanan Subramanian < > gokul24...@

Re: Why should SASL principal be unchanged upon reauth

2020-09-28 Thread Gokul Ramanan Subramanian
ino wrote: > Hi Gokul. I looked back at the discussion thread, and it appears it was an > arbitrary decision. > > > https://lists.apache.org/thread.html/45c09f226386c0b1dc5f9b36e112882a20414d5900f8d778969e633e%40%3Cdev.kafka.apache.org%3E > > Ron > > On Thu, Sep 24, 2020

Why should SASL principal be unchanged upon reauth

2020-09-24 Thread Gokul Ramanan Subramanian
Hi. I was looking through Kafka code and found that SASL KafkaPrincipals are not supposed to change upon reauthentication, and if they do, the broker will kill the TCP connection. What is the reasoning behind this limitation? Thanks.

Re: [VOTE] KIP-578: Add configuration to limit number of partitions

2020-09-21 Thread Gokul Ramanan Subramanian
, Aug 5, 2020 at 5:59 PM Harsha Ch wrote: > Thanks for the updates Gokul. +1 (binding). > > Thanks, > > Harsha > > On Wed, Aug 05, 2020 at 8:18 AM, Gokul Ramanan Subramanian < > gokul24...@gmail.com > wrote: > > > > > > > > > Hi. > >

Re: [DISCUSS] KIP-516: Topic Identifiers

2020-09-11 Thread Gokul Ramanan Subramanian
Hi. Thanks for the KIP. Have you thought about whether it makes sense to support authorizing a principal for a topic ID rather than a topic name to achieve tighter security? Or is the topic ID fundamentally an internal detail similar to epochs used in a bunch of other places in Kafka? Thanks.

Re: [VOTE] KIP-578: Add configuration to limit number of partitions

2020-08-05 Thread Gokul Ramanan Subramanian
Hi. I request the community to kindly resume the voting process for KIP-578. If you have further comments, we can address those as well. Thanks and cheers. On Tue, Jul 21, 2020 at 2:07 PM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Hi. > > Can we resume the

Re: [VOTE] KIP-578: Add configuration to limit number of partitions

2020-07-21 Thread Gokul Ramanan Subramanian
Hi. Can we resume the voting process for KIP-578? I have addressed additional comments by Boyang and Ismael. Thanks. On Mon, Jun 8, 2020 at 9:09 AM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Hi. Can we resume the voting process for KIP-578? Thanks. > > On Mon, Ju

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-07-14 Thread Gokul Ramanan Subramanian
t; > described there on both replication overhead and topic creation/deletion > > performance. > > > > Ismael > > > > On Tue, Jun 23, 2020 at 11:10 PM Gokul Ramanan Subramanian < > > gokul24...@gmail.com> wrote: > > > > > Ismael, > > > >

Re: Running system tests on mac

2020-06-30 Thread Gokul Ramanan Subramanian
> > best, > Colin > > On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian wrote: > > Hi. > > > > Has anyone had luck running Kafka system tests on a Mac. I have a MacOS > > Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the command > > *duc

Re: Running system tests on mac

2020-06-29 Thread Gokul Ramanan Subramanian
4GB of RAM free and between 15-45 % idle CPU. So, I don't think I am running low on resources. Is this common? At this rate, the tests would take about 2 days to complete and there'd probably be lots of failures. On Mon, Jun 29, 2020 at 11:21 AM Gokul Ramanan Subramanian < gokul24...@gmail.com> wr

Re: Running system tests on mac

2020-06-29 Thread Gokul Ramanan Subramanian
instruction on [1] and it works like a charm. > > [1] > https://github.com/apache/kafka/tree/trunk/tests#running-tests-using-docker > > > > 29 июня 2020 г., в 12:14, Gokul Ramanan Subramanian < > gokul24...@gmail.com> написал(а): > > > > Hi. > > &g

Running system tests on mac

2020-06-29 Thread Gokul Ramanan Subramanian
Hi. Has anyone had luck running Kafka system tests on a Mac. I have a MacOS Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the command *ducktape tests/kafkatest/tests* yields the following error, making it look like some Python incompatibility issue. $ ducktape tests/kafkatest/tests

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

2020-06-24 Thread Gokul Ramanan Subramanian
in 2014). > > Gwen > > > On Wed, Jun 24, 2020 at 10:40 AM Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > > > I agree that is an option, but is there any reason to not have a 2.3.2 > and > > 2.4.2 released? If so, it would be nice to know abou

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

2020-06-24 Thread Gokul Ramanan Subramanian
5.0 and > completely ignore ZK and TLS. It's completely optional. > > Ismael > > On Wed, Jun 24, 2020 at 10:20 AM Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > > > Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of migrating > > from non

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

2020-06-24 Thread Gokul Ramanan Subramanian
Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of migrating from non-encrypted TLS to encrypted TLS connection for ZooKeeper, it would be nice to have a bug-free version of 2.3 and 2.4. Is there a technical reason why we hesitate to get these versions out? Or is it that no one has got

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-06-24 Thread Gokul Ramanan Subramanian
about > framework-level topics from other tools and extensions? (connect, > streams, confluent metrics, tiered storage, etc.) Is blacklisting > possible? > > 907. What happens if one of the dynamic limit is violated at update > time? (sorry if it's already explained in the KIP,

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-06-24 Thread Gokul Ramanan Subramanian
onnect, > > streams, confluent metrics, tiered storage, etc.) Is blacklisting > > possible? > > > > 907. What happens if one of the dynamic limit is violated at update > > time? (sorry if it's already explained in the KIP, may have missed it) > > > > Thanks,

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-06-24 Thread Gokul Ramanan Subramanian
perf section? I think master should do > better than what's described there. > > Ismael > > On Wed, Apr 1, 2020, 8:28 AM Gokul Ramanan Subramanian < > gokul24...@gmail.com> > wrote: > > > Hi. > > > > I have opened KIP-578, intended to provide a mechanis

Re: [VOTE] KIP-578: Add configuration to limit number of partitions

2020-06-08 Thread Gokul Ramanan Subramanian
Hi. Can we resume the voting process for KIP-578? Thanks. On Mon, Jun 1, 2020 at 11:09 AM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Thanks Colin. Have updated the KIP per your recommendations. Let me know > what you think. > > Thanks Harsha for the vote. > &

Re: [VOTE] KIP-578: Add configuration to limit number of partitions

2020-06-01 Thread Gokul Ramanan Subramanian
Thanks Colin. Have updated the KIP per your recommendations. Let me know what you think. Thanks Harsha for the vote. On Wed, May 27, 2020 at 8:17 PM Colin McCabe wrote: > Hi Gokul Ramanan Subramanian, > > Thanks for the KIP. > > Can you please modify the KIP to remov

Re: [VOTE] KIP-578: Add configuration to limit number of partitions

2020-05-26 Thread Gokul Ramanan Subramanian
Hi. Any votes for this? Thanks. On Tue, May 12, 2020 at 11:36 AM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Hello, > > I'd like to initialize voting on KIP-578: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+nu

[VOTE] KIP-578: Add configuration to limit number of partitions

2020-05-12 Thread Gokul Ramanan Subramanian
Hello, I'd like to initialize voting on KIP-578: https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions . Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez and Tom Bentley on the discussion thread. I have addressed their

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-05-03 Thread Gokul Ramanan Subramanian
y, yet a new > release of Kafka introduces a new topic (e.g KIP-500)? > > Best, > Stanislav > > On Fri, Apr 24, 2020 at 2:39 PM Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > > > Hi Tom. > > > > With KIP-578, we are not trying to model th

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-04-24 Thread Gokul Ramanan Subramanian
values for these configs (both on day 1 and in an > evolving production cluster), when this solution might be better than using > a cluster balancer and/or why cluster balancers can't be trusted to avoid > overloading brokers. > > Kind regards, > > Tom > > > On

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-04-20 Thread Gokul Ramanan Subramanian
the work to make all ZK writes happen on the > controller. > > Of course, this is just my take on it. > > Kind regards, > > Tom > > On Thu, Apr 16, 2020 at 11:47 AM Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > > > Hi. > > > >

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-04-16 Thread Gokul Ramanan Subramanian
their thoughts, so that I can more confidently start the voting thread. Thanks. On Thu, Apr 16, 2020 at 11:30 AM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Thanks for your comments Alex. > > The KIP proposes using two configurations max.partitions and > max.broker.p

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-04-16 Thread Gokul Ramanan Subramanian
aining the objective function and > reducing the migration path. > > Thanks, > Alexandre > > Le jeu. 9 avr. 2020 à 00:19, Gokul Ramanan Subramanian > a écrit : > > > > Hi. Requesting you to take a look at this KIP and provide feedback. > > > > Thanks. R

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-04-08 Thread Gokul Ramanan Subramanian
Hi. Requesting you to take a look at this KIP and provide feedback. Thanks. Regards. On Wed, Apr 1, 2020 at 4:28 PM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Hi. > > I have opened KIP-578, intended to provide a mechanism to limit the number > of partitions

[DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-04-01 Thread Gokul Ramanan Subramanian
Hi. I have opened KIP-578, intended to provide a mechanism to limit the number of partitions in a Kafka cluster. Kindly provide feedback on the KIP which you can find at https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions I want to

Re: [VOTE] KIP-579: new exception on min.insync.replicas > replication.factor

2020-03-31 Thread Gokul Ramanan Subramanian
+1 (non binding). Thanks. On Tue, Mar 31, 2020 at 2:43 PM Alexandre Dupriez < alexandre.dupr...@gmail.com> wrote: > +1 (non binding). > > Thanks. > > Le mar. 31 mars 2020 à 14:24, M. Manna a écrit : > > > > +1 (binding). > > > > Thanks for the KIP. > > > > On Tue, 31 Mar 2020 at 14:17, Paolo

Has restarted broker caught up?

2020-03-11 Thread Gokul Ramanan Subramanian
Hi. What is the idiomatic / correct way to know that a restarted broker in an existing Kafka cluster has caught up to all the leaders for the partitions it hosts? Is there a specific JMX metric published either by the leader / follower that we can monitor? Ideally, the metric would have the

Limit number of topic partitions

2020-03-06 Thread Gokul Ramanan Subramanian
Hi. I'd like your thoughts on https://issues.apache.org/jira/browse/KAFKA-9590?jql=text%20~%20%22limit%20topic%20partitions%22 (tldr: its about limiting number of topic partitions via a read-only configuration). I am going to start writing a KIP for this soon, and will share on this thread as I

[jira] [Created] (KAFKA-9590) Add read-only configuration to limit number of topic partitions

2020-02-21 Thread Gokul Ramanan Subramanian (Jira)
Gokul Ramanan Subramanian created KAFKA-9590: Summary: Add read-only configuration to limit number of topic partitions Key: KAFKA-9590 URL: https://issues.apache.org/jira/browse/KAFKA-9590

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-12-20 Thread Gokul Ramanan Subramanian
Application > > >> A, but > > >> > > because it has a faster processing time its latency per partition > is > > >> > > significantly less. > > >> > > > > >> > > If the Kafka Consumer reported a latency metric it would be > easier to > > >> > > build Service Level Agreements (SLAs) based on non-functional > > >> requirements > > >> > > of the streaming system. For example, the system must never have a > > >> latency > > >> > > of greater than 10 minutes. This SLA could be used in monitoring > > >> alerts or > > >> > > as input to automatic scaling solutions. > > >> > > > > >> > > On Thu, Jul 11, 2019 at 12:36 PM Sean Glover < > > >> sean.glo...@lightbend.com> > > >> > > wrote: > > >> > > > > >> > >> Hi kafka-dev, > > >> > >> > > >> > >> I've created KIP-489 as a proposal for adding latency metrics to > the > > >> > >> Kafka Consumer in a similar way as record-lag metrics are > > >> implemented. > > >> > >> > > >> > >> > > >> > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/489%3A+Kafka+Consumer+Record+Latency+Metric > > >> > >> > > >> > >> Regards, > > >> > >> Sean > > >> > >> > > >> > >> -- > > >> > >> Principal Engineer, Lightbend, Inc. > > >> > >> > > >> > >> <http://lightbend.com> > > >> > >> > > >> > >> @seg1o <https://twitter.com/seg1o>, in/seanaglover > > >> > >> <https://www.linkedin.com/in/seanaglover/> > > >> > >> > > >> > > > > >> > > > > >> > > -- > > >> > > Principal Engineer, Lightbend, Inc. > > >> > > > > >> > > <http://lightbend.com> > > >> > > > > >> > > @seg1o <https://twitter.com/seg1o>, in/seanaglover > > >> > > <https://www.linkedin.com/in/seanaglover/> > > >> > > > > >> > > > >> > > > > > > > > > > -- > > Sean Glover > > Principal Engineer, Alpakka, Lightbend, Inc. <https://lightbend.com> > > @seg1o <https://twitter.com/seg1o>, in/seanaglover > > <https://www.linkedin.com/in/seanaglover/> > > > -- Gokul Ramanan Subramanian Senior SDE, Amazon AWS

Re: [DISCUSS] KIP-498: Add client-side configuration for maximum response size to protect against OOM

2019-08-06 Thread Gokul Ramanan Subramanian
avax.net.ssl.SSLException: Inbound closed before > > receiving peer's close_notify: possible truncation attack? > > > > [Raw write]: length = 7 > > > > : 15 03 03 00 02 02 50 ..P > > > > > > Le jeu. 1 août 2019

Re: [DISCUSS] KIP-498: Add client-side configuration for maximum response size to protect against OOM

2019-07-31 Thread Gokul Ramanan Subramanian
if my +1 counts, since I am a Kafka noob). Thanks. Gokul Ramanan Subramanian Senior SDE, Amazon AWS On 28/Jul/19 05:43:19PM +0100, Alexandre Dupriez wrote: Hello, I have created the KIP-498 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maxi

JIRA and KIP contributor permissions

2019-07-30 Thread Gokul Ramanan Subramanian
Hello Community, In order to start contributing to Apache Kafka project, could I please request contributor access to JIRA and be granted write permissions to the Kafka wiki? My name: Gokul Ramanan Subramanian JIRA username: gokul2411s Confluence username: gokul2411s (https://cwiki.apache.org