Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

[Jeff Chao]

Hi Jun, same for me. jeffc...@me.com

Thanks,
Jeff Chao

On Thu, Apr 5, 2018 at 9:06 PM, Jun Rao  wrote:

> Hi, James, Jeff,
>
> Added you to the invitation.
>
> Jun
>
> On Thu, Apr 5, 2018 at 6:11 PM, Jeff Widman  wrote:
>
> > Please add me as well: j...@jeffwidman.com
> >
> > thanks
> >
> > On Thu, Apr 5, 2018 at 5:16 PM, James Cheng 
> wrote:
> >
> > > Jun,
> > >
> > > Can you add me as well? wushuja...@gmail.com  > wushuja...@gmail.com>
> > >
> > > Thanks,
> > > -James
> > >
> > > > On Apr 5, 2018, at 1:34 PM, Jun Rao  wrote:
> > > >
> > > > Hi, Ted, Vahid,
> > > >
> > > > Added you to the invite.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Thu, Apr 5, 2018 at 10:42 AM, Vahid S Hashemian <
> > > > vahidhashem...@us.ibm.com> wrote:
> > > >
> > > >> Hi Jun,
> > > >>
> > > >> I used to receive these invites, but didn't get this one.
> > > >> Please send me an invite. Thanks.
> > > >>
> > > >> Regards,
> > > >> --Vahid
> > > >>
> > > >>
> > > >>
> > > >> From:   Jun Rao 
> > > >> To: dev 
> > > >> Date:   04/05/2018 10:25 AM
> > > >> Subject:Kafka KIP meeting on Apr. 9 at 9:00am PDT
> > > >>
> > > >>
> > > >>
> > > >> Hi, Everyone,
> > > >>
> > > >> We plan to have a Kafka KIP meeting this coming Monday (Apr. 9)  at
> > > 9:00am
> > > >> PDT. If you plan to attend but haven't received an invite, please
> let
> > me
> > > >> know. The following is the agenda.
> > > >>
> > > >> Agenda:
> > > >> KIP-253: Partition expansion
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Jun
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > >
> > >
> >
> >
> > --
> >
> > *Jeff Widman*
> > jeffwidman.com  | 740-WIDMAN-J (943-6265)
> > <><
> >
>

[jira] [Created] (KAFKA-6756) client.id and group.id validation in the old vs new consumer

[Badai Aqrandista (JIRA)]

Badai Aqrandista created KAFKA-6756:
---

 Summary: client.id and group.id validation in the old vs new 
consumer
 Key: KAFKA-6756
 URL: https://issues.apache.org/jira/browse/KAFKA-6756
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 1.1.0, 0.10.2.1
Reporter: Badai Aqrandista


It looks like the old consumer that is based on the Scala code validates 
"client.id" and "group.id" using this code:

[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/common/Config.scala#L25]

[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/consumer/ConsumerConfig.scala#L60]

 

However, the new consumer uses the Java code that does not validate "client.id" 
and "group.id" at all here:

[https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java#L264]

[https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java#L298]

So the new consumer does not validate "client.id" and "group.id" like the old 
consumer. Either way, the documentation never specify the valid character for 
"client.id" and "group.id".

Is this a bug or by design?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

[Jun Rao]

Hi, James, Jeff,

Added you to the invitation.

Jun

On Thu, Apr 5, 2018 at 6:11 PM, Jeff Widman  wrote:

> Please add me as well: j...@jeffwidman.com
>
> thanks
>
> On Thu, Apr 5, 2018 at 5:16 PM, James Cheng  wrote:
>
> > Jun,
> >
> > Can you add me as well? wushuja...@gmail.com  wushuja...@gmail.com>
> >
> > Thanks,
> > -James
> >
> > > On Apr 5, 2018, at 1:34 PM, Jun Rao  wrote:
> > >
> > > Hi, Ted, Vahid,
> > >
> > > Added you to the invite.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Apr 5, 2018 at 10:42 AM, Vahid S Hashemian <
> > > vahidhashem...@us.ibm.com> wrote:
> > >
> > >> Hi Jun,
> > >>
> > >> I used to receive these invites, but didn't get this one.
> > >> Please send me an invite. Thanks.
> > >>
> > >> Regards,
> > >> --Vahid
> > >>
> > >>
> > >>
> > >> From:   Jun Rao 
> > >> To: dev 
> > >> Date:   04/05/2018 10:25 AM
> > >> Subject:Kafka KIP meeting on Apr. 9 at 9:00am PDT
> > >>
> > >>
> > >>
> > >> Hi, Everyone,
> > >>
> > >> We plan to have a Kafka KIP meeting this coming Monday (Apr. 9)  at
> > 9:00am
> > >> PDT. If you plan to attend but haven't received an invite, please let
> me
> > >> know. The following is the agenda.
> > >>
> > >> Agenda:
> > >> KIP-253: Partition expansion
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >>
> > >>
> > >>
> > >>
> >
> >
>
>
> --
>
> *Jeff Widman*
> jeffwidman.com  | 740-WIDMAN-J (943-6265)
> <><
>

Re: [VOTE] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

[Guozhang Wang]

Thanks for the KIP!

+1 (binding)

On Thu, Apr 5, 2018 at 3:11 PM, Matthias J. Sax 
wrote:

> +1 (binding)
>
>
> -Matthias
>
> On 4/5/18 4:36 AM, Ted Yu wrote:
> > +1
> >  Original message From: Mickael Maison <
> mickael.mai...@gmail.com> Date: 4/5/18  1:42 AM  (GMT-08:00) To: dev <
> dev@kafka.apache.org> Subject: Re: [VOTE] KIP-211: Revise Expiration
> Semantics of Consumer Group Offsets
> > +1 (non-binding)
> > Thanks for the KIP!
> >
> > On Thu, Apr 5, 2018 at 8:08 AM, Jason Gustafson 
> wrote:
> >> +1 Thanks Vahid!
> >>
> >> On Wed, Mar 28, 2018 at 7:27 PM, James Cheng 
> wrote:
> >>
> >>> +1 (non-binding)
> >>>
> >>> Thanks for all the hard work on this, Vahid!
> >>>
> >>> -James
> >>>
>  On Mar 28, 2018, at 10:34 AM, Vahid S Hashemian <
> >>> vahidhashem...@us.ibm.com> wrote:
> 
>  Hi all,
> 
>  As I believe the feedback and suggestions on this KIP have been
> addressed
>  so far, I'd like to start a vote.
>  The KIP can be found at
>  https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> 211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets
> 
>  Thanks in advance for voting :)
> 
>  --Vahid
> 
> >>>
> >>>
>
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

[Boyang Chen]

Hey guys,


thanks for the discussion. One good point Matthias pointed out was that the 
whole "consumer config separation" is for advanced users. In case someone wants 
to set different consumer configs, he must already be searching the codebase to 
figure out how to do that; if a user doesn't want to set different consumer 
type configs, he would follow the same "consumer." prefix to change the base. 
This should cause little confusion to entry level users. (at least he has to 
figure out what global/restore consumers are)


Best,

Boyang


From: Matthias J. Sax 
Sent: Friday, April 6, 2018 6:28 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

Ewen,

I cannot completely follow your argument. Can you elaborate a little
bit? After reading you mail, I am not sure if you prefer config
inheritance or not? And if, to what extend?


> In that mode if you override *anything*, you
>> should specify *everything*.

Can you give an example for this?


> But if it was just, e.g. group.id, client.id,
>> and offset.reset that needed adjustment for a restoreConsumer, that would
>> be the default and everything else is inherited.

Don't understand this part.


> I think
>> inheritance in these configuration setups needs to be approached very
>> carefully.

Agreed. I think that our approach is carefully designed though.


For follow up on Guozhang's argument, I agree with him, that for most
users the existing prefix would be good enough and the three new
prefixes are for advanced users.



-Matthias

On 4/5/18 11:37 AM, Guozhang Wang wrote:
> Ewen, thanks for your thoughts.
>
> Just to clarify the current KIP does not propose for four new prefixes, but
> three plus the existing one. So it is
>
> "consumer"
> "main.consumer"
> "restore.consumer"
> "global.consumer"
>
>
> If we design the configs for the first time, I'd be in favor of only
> keeping the last three prefixes. But as of now we have a compatibility
> issue to consider. So the question is if it is worthwhile to break the
> compatibility and enforce users to make code changes. My rationale is that:
>
> 1) for normal users they would not bother overriding configs for different
> types of consumers, where "consumer" prefix is good enough for them; and
> today they probably have already made those overrides via "consumer".
>
> 2) for advanced users they would need some additional overrides for
> different types of consumers, and they would go ahead and learn about the
> other three prefixes and set them there.
>
>
> I agree that four prefixes would be more confusing, but if we think use
> case 1)'s popularity is much larger than use case 2), which by the way we
> can still debate on, then I'd argue it's better to not force normal user
> groups from 1) to make code changes to make advanced users from 2) less
> confused about the hierarchy.
>
>
> Guozhang
>
>
>
>
>
>
> On Wed, Apr 4, 2018 at 11:23 PM, Ewen Cheslack-Postava 
> wrote:
>
>> I think this model is more confusing than it needs to be.
>>
>> We end up with 4 prefixes despite only have 3 types of consumers. We have
>> prefixes for: "base", "main", "global", and "restore". However, we only
>> instantiate consumers of type "main", "global", and "restore".
>>
>> Until now, we've only had two types of configs mapping to two types of
>> consumers, despite internally having some common shared configs as a
>> baseline to bootstrap the two "public" ones (see
>> StreamsConfig.getCommonConsumerConfigs). Do we want to complicate this to
>> 4
>> levels of "public" configs when there are only 3 types of concrete configs
>> we instantiate?
>>
>> More generally, I worry that we're optimizing too much to avoid copy/paste
>> in less common cases to the point that we would confuse users with yet more
>> concepts before they can even write their configuration. What if we took an
>> (perhaps modified) all or nothing approach to inheriting from the the
>> "main" consumer properties? In that mode if you override *anything*, you
>> should specify *everything*. But if it was just, e.g. group.id, client.id,
>> and offset.reset that needed adjustment for a restoreConsumer, that would
>> be the default and everything else is inherited. Same deal for a clearly
>> specified set of configs for global consumers that required override.
>>
>> I feel like I'm also concurrently seeing the opposite side of this problem
>> in Connect where we namespaced and didn't proactively implement
>> inheritance; and while people find the config duplication annoying (and
>> confusing!), we inevitably find cases where they need it. I think
>> inheritance in these configuration setups needs to be approached very
>> carefully. Admittedly, some of the challenges in Connect don't appear here
>> (e.g. conflicts in producer/consumer config naming, since this is a
>> Consumer-only KIP), but similar problems arise.
>>
>> -Ewen
>>
>> 

Jenkins build is back to normal : kafka-1.1-jdk7 #113

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null

[Randall Hauch (JIRA)]

Randall Hauch created KAFKA-6755:


 Summary: MaskField SMT should optionally take a literal value to 
use instead of using null
 Key: KAFKA-6755
 URL: https://issues.apache.org/jira/browse/KAFKA-6755
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Affects Versions: 0.11.0.0
Reporter: Randall Hauch


The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always uses 
the null value for the type of field. It'd be nice to *optionally* be able to 
specify a literal value for the type, where the SMT would convert the literal 
string value in the configuration to the desired type (using the new {{Values}} 
methods).

Use cases: mask out the IP address, or SSN, or other personally identifiable 
information (PII).

Since this changes the API, and thus will require a KIP.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

[Jeff Widman]

Please add me as well: j...@jeffwidman.com

thanks

On Thu, Apr 5, 2018 at 5:16 PM, James Cheng  wrote:

> Jun,
>
> Can you add me as well? wushuja...@gmail.com 
>
> Thanks,
> -James
>
> > On Apr 5, 2018, at 1:34 PM, Jun Rao  wrote:
> >
> > Hi, Ted, Vahid,
> >
> > Added you to the invite.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Apr 5, 2018 at 10:42 AM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> >
> >> Hi Jun,
> >>
> >> I used to receive these invites, but didn't get this one.
> >> Please send me an invite. Thanks.
> >>
> >> Regards,
> >> --Vahid
> >>
> >>
> >>
> >> From:   Jun Rao 
> >> To: dev 
> >> Date:   04/05/2018 10:25 AM
> >> Subject:Kafka KIP meeting on Apr. 9 at 9:00am PDT
> >>
> >>
> >>
> >> Hi, Everyone,
> >>
> >> We plan to have a Kafka KIP meeting this coming Monday (Apr. 9)  at
> 9:00am
> >> PDT. If you plan to attend but haven't received an invite, please let me
> >> know. The following is the agenda.
> >>
> >> Agenda:
> >> KIP-253: Partition expansion
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >>
> >>
> >>
>
>


-- 

*Jeff Widman*
jeffwidman.com  | 740-WIDMAN-J (943-6265)
<><

Jenkins build is back to normal : kafka-trunk-jdk9 #537

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-6754) Allow Kafka to be used for horizontally-scalable real-time stream visualization

[Bill DeStein (JIRA)]

Bill DeStein created KAFKA-6754:
---

 Summary: Allow Kafka to be used for horizontally-scalable 
real-time stream visualization
 Key: KAFKA-6754
 URL: https://issues.apache.org/jira/browse/KAFKA-6754
 Project: Kafka
  Issue Type: New Feature
  Components: clients
Reporter: Bill DeStein


I've developed a patch that allows Kafka to be used as the back-end for 
horizontally-scalable real-time stream visualization systems.

I've created a five-minute demo video here.

[http://billdestein.software.s3.amazonaws.com/sitegrapher.html]

I'd like to get thought from the Kafka leadership on whether my patch could be 
made part of Kafka going forward.  I'll create a wiki with implementation 
details if there is interest.

I intend to open source the time series portal as a separate project because it 
will be lots of React and Redux code that probably doesn't belong in the Kafka 
code base.

Thanks,  Bill DeStein



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6753) Speed up event processing on the controller

[Lucas Wang (JIRA)]

Lucas Wang created KAFKA-6753:
-

 Summary: Speed up event processing on the controller 
 Key: KAFKA-6753
 URL: https://issues.apache.org/jira/browse/KAFKA-6753
 Project: Kafka
  Issue Type: Improvement
Reporter: Lucas Wang
Assignee: Lucas Wang
 Attachments: Screen Shot 2018-04-04 at 7.08.55 PM.png

The existing controller code updates metrics after processing every event. This 
can slow down event processing on the controller tremendously. In one profiling 
we see that updating metrics takes nearly 100% of the CPU for the controller 
event processing thread. Specifically the slowness can be attributed to two 
factors:
1. Each invocation to update the metrics is expensive. Specifically trying to 
calculate the offline partitions count requires iterating through all the 
partitions in the cluster to check if the partition is offline; and calculating 
the preferred replica imbalance count requires iterating through all the 
partitions in the cluster to check if a partition has a leader other than the 
preferred leader. In a large cluster, the number of partitions can be quite 
large, all seen by the controller. Even if the time spent to check a single 
partition is small, the accumulation effect of so many partitions in the 
cluster can make the invocation to update metrics quite expensive. One might 
argue that maybe the logic for processing each single partition is not 
optimized, we checked the CPU percentage of leaf nodes in the profiling result, 
and found that inside the loops of collection objects, e.g. the set of all 
partitions, no single function dominates the processing. Hence the large number 
of the partitions in a cluster is the main contributor to the slowness of one 
invocation to update the metrics.
2. The invocation to update metrics is called many times when the is a high 
number of events to be processed by the controller, one invocation after 
processing any event.

The patch that will be submitted tries to fix bullet 2 above, i.e. reducing the 
number of invocations to update metrics. Instead of updating the metrics after 
processing any event, we only periodically check if the metrics needs to be 
updated, i.e. once every second. 
* If after the previous invocation to update metrics, there are other types of 
events that changed the controller’s state, then one second later the metrics 
will be updated. 
* If after the previous invocation, there has been no other types of events, 
then the call to update metrics can be bypassed.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

[James Cheng]

Jun, 

Can you add me as well? wushuja...@gmail.com 

Thanks,
-James

> On Apr 5, 2018, at 1:34 PM, Jun Rao  wrote:
> 
> Hi, Ted, Vahid,
> 
> Added you to the invite.
> 
> Thanks,
> 
> Jun
> 
> 
> On Thu, Apr 5, 2018 at 10:42 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
> 
>> Hi Jun,
>> 
>> I used to receive these invites, but didn't get this one.
>> Please send me an invite. Thanks.
>> 
>> Regards,
>> --Vahid
>> 
>> 
>> 
>> From:   Jun Rao 
>> To: dev 
>> Date:   04/05/2018 10:25 AM
>> Subject:Kafka KIP meeting on Apr. 9 at 9:00am PDT
>> 
>> 
>> 
>> Hi, Everyone,
>> 
>> We plan to have a Kafka KIP meeting this coming Monday (Apr. 9)  at 9:00am
>> PDT. If you plan to attend but haven't received an invite, please let me
>> know. The following is the agenda.
>> 
>> Agenda:
>> KIP-253: Partition expansion
>> 
>> Thanks,
>> 
>> Jun
>> 
>> 
>> 
>> 
>> 

Re: [DISCUSS] KIP-257 - Configurable Quota Management

[Ismael Juma]

Hi Rajini,

Can you share the motivation for the change? Not sure if the new approach
is better.

Ismael

On Thu, Apr 5, 2018 at 4:06 PM, Rajini Sivaram 
wrote:

> The quota callback interface was updated based on Jun's suggestion during
> the PR review. I have updated the KIP to reflect this.
>
> Let me know if there are any concerns.
>
> Thanks,
>
> Rajini
>
> On Thu, Apr 5, 2018 at 1:04 PM, Rajini Sivaram 
> wrote:
>
> > Thanks, Ismael.
> >
> > I have updated the KIP and the PR.
> >
> > On Wed, Apr 4, 2018 at 7:37 PM, Ismael Juma  wrote:
> >
> >> Sounds good to me Rajini. Good catch spotting this before it's included
> in
> >> a release. :)
> >>
> >> Ismael
> >>
> >> On Wed, Apr 4, 2018 at 11:13 AM, Rajini Sivaram <
> rajinisiva...@gmail.com>
> >> wrote:
> >>
> >> > For compatibility reasons, we are now using Java rather than Scala for
> >> all
> >> > pluggable interfaces including those on the broker. There is already a
> >> KIP
> >> > to move Authorizer to Java as well. As we will be removing support for
> >> Java
> >> > 7 in the next release, we can also use default methods in Java when we
> >> need
> >> > to update pluggable Java interfaces. So the plan is to use Java for
> all
> >> new
> >> > pluggable interfaces.
> >> >
> >> > We already have the package org.apache.kafka.server, under which we
> have
> >> > the sub-package for policies, so it makes sense to define quota
> >> callback as
> >> > a Java interface here too.
> >> >
> >> > If there are any concerns, please let me know. Otherwise I will update
> >> the
> >> > KIP and the associated PR.
> >> >
> >> > Thank you,
> >> >
> >> > Rajini
> >> >
> >> > On Thu, Mar 22, 2018 at 9:52 PM, Rajini Sivaram <
> >> rajinisiva...@gmail.com>
> >> > wrote:
> >> >
> >> > > Since there all the comments so far have been addressed, I will
> start
> >> > vote
> >> > > for this KIP.
> >> > >
> >> > > Regards,
> >> > >
> >> > > Rajini
> >> > >
> >> > > On Thu, Mar 15, 2018 at 6:37 PM, Rajini Sivaram <
> >> rajinisiva...@gmail.com
> >> > >
> >> > > wrote:
> >> > >
> >> > >> Thanks, Jun.
> >> > >>
> >> > >> 11. updatePartitionMetadata() provides all partitions with their
> >> leaders
> >> > >> so that callbacks that scale down quotas based on fraction of
> >> partitions
> >> > >> hosted on each broker may compute the scaling factor. Callbacks
> that
> >> > scale
> >> > >> up quotas based on the partition count hosted on each broker can
> >> simply
> >> > >> filter out the others. I have updated the Javadoc in the KIP.
> >> > >>
> >> > >> On Thu, Mar 15, 2018 at 5:24 PM, Jun Rao  wrote:
> >> > >>
> >> > >>> Hi, Rajini,
> >> > >>>
> >> > >>> Thanks for the explanation. It makes sense.
> >> > >>>
> >> > >>> 11. We probably want to clarify in the interface that every time
> >> when
> >> > >>> updatePartitionMetadata() is called, the full list of partitions
> >> whose
> >> > >>> leader is on this broker will be passed in?
> >> > >>>
> >> > >>> Jun
> >> > >>>
> >> > >>> On Thu, Mar 15, 2018 at 4:42 AM, Rajini Sivaram <
> >> > rajinisiva...@gmail.com
> >> > >>> >
> >> > >>> wrote:
> >> > >>>
> >> > >>> > Hi Jun,
> >> > >>> >
> >> > >>> > 12. Sorry, I had to revert the change that removed `
> >> > >>> > ClientQuotaCallback.quotaLimit()`. We are allowing quota
> >> callbacks
> >> > to
> >> > >>> use
> >> > >>> > custom metric tags. For each request, quota manager uses `
> >> > >>> > ClientQuotaCallback.quota()` to map (user-principal, client-id)
> to
> >> > the
> >> > >>> > metric tags that determine which clients share the quota. When
> >> quotas
> >> > >>> are
> >> > >>> > updated using  `updateQuota` or `updatePartitionMetadata`,
> >> existing
> >> > >>> metrics
> >> > >>> > need to updated, but quota managers don't have a reverse mapping
> >> of
> >> > >>> metric
> >> > >>> > tags to (user-principal, client-id) for
> >> invoking`ClientQuotaCallback.
> >> > >>> > quota()
> >> > >>> > ` . Callbacks cannot return all updated metrics since they don't
> >> have
> >> > >>> > access to the metrics object and we don't want to require
> >> callbacks
> >> > to
> >> > >>> > track all the entities for which metrics have been created
> (since
> >> > they
> >> > >>> may
> >> > >>> > contain client-ids and hence need expiring). With the extra
> >> method,
> >> > >>> quota
> >> > >>> > manager traverses the metric list after `updateQuota` or `
> >> > >>> > updatePartitionMetadata` and obtains the latest value
> >> corresponding
> >> > to
> >> > >>> each
> >> > >>> > metric based on the tags using `ClientQuotaCallback.quotaLimi
> >> t()`.
> >> > >>> >
> >> > >>> > An alternative may be to delay quota metrics updates until the
> >> next
> >> > >>> request
> >> > >>> > that uses the metric. When we get sensors, we can check if the
> >> quota
> >> > >>> > configured in the metric matches the value returned by `
> >> > >>> > ClientQuotaCallback.quota()`. This will be slightly more
> expensive
> >> > >>> 

Jenkins build is back to normal : kafka-1.0-jdk7 #184

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-257 - Configurable Quota Management

[Rajini Sivaram]

The quota callback interface was updated based on Jun's suggestion during
the PR review. I have updated the KIP to reflect this.

Let me know if there are any concerns.

Thanks,

Rajini

On Thu, Apr 5, 2018 at 1:04 PM, Rajini Sivaram 
wrote:

> Thanks, Ismael.
>
> I have updated the KIP and the PR.
>
> On Wed, Apr 4, 2018 at 7:37 PM, Ismael Juma  wrote:
>
>> Sounds good to me Rajini. Good catch spotting this before it's included in
>> a release. :)
>>
>> Ismael
>>
>> On Wed, Apr 4, 2018 at 11:13 AM, Rajini Sivaram 
>> wrote:
>>
>> > For compatibility reasons, we are now using Java rather than Scala for
>> all
>> > pluggable interfaces including those on the broker. There is already a
>> KIP
>> > to move Authorizer to Java as well. As we will be removing support for
>> Java
>> > 7 in the next release, we can also use default methods in Java when we
>> need
>> > to update pluggable Java interfaces. So the plan is to use Java for all
>> new
>> > pluggable interfaces.
>> >
>> > We already have the package org.apache.kafka.server, under which we have
>> > the sub-package for policies, so it makes sense to define quota
>> callback as
>> > a Java interface here too.
>> >
>> > If there are any concerns, please let me know. Otherwise I will update
>> the
>> > KIP and the associated PR.
>> >
>> > Thank you,
>> >
>> > Rajini
>> >
>> > On Thu, Mar 22, 2018 at 9:52 PM, Rajini Sivaram <
>> rajinisiva...@gmail.com>
>> > wrote:
>> >
>> > > Since there all the comments so far have been addressed, I will start
>> > vote
>> > > for this KIP.
>> > >
>> > > Regards,
>> > >
>> > > Rajini
>> > >
>> > > On Thu, Mar 15, 2018 at 6:37 PM, Rajini Sivaram <
>> rajinisiva...@gmail.com
>> > >
>> > > wrote:
>> > >
>> > >> Thanks, Jun.
>> > >>
>> > >> 11. updatePartitionMetadata() provides all partitions with their
>> leaders
>> > >> so that callbacks that scale down quotas based on fraction of
>> partitions
>> > >> hosted on each broker may compute the scaling factor. Callbacks that
>> > scale
>> > >> up quotas based on the partition count hosted on each broker can
>> simply
>> > >> filter out the others. I have updated the Javadoc in the KIP.
>> > >>
>> > >> On Thu, Mar 15, 2018 at 5:24 PM, Jun Rao  wrote:
>> > >>
>> > >>> Hi, Rajini,
>> > >>>
>> > >>> Thanks for the explanation. It makes sense.
>> > >>>
>> > >>> 11. We probably want to clarify in the interface that every time
>> when
>> > >>> updatePartitionMetadata() is called, the full list of partitions
>> whose
>> > >>> leader is on this broker will be passed in?
>> > >>>
>> > >>> Jun
>> > >>>
>> > >>> On Thu, Mar 15, 2018 at 4:42 AM, Rajini Sivaram <
>> > rajinisiva...@gmail.com
>> > >>> >
>> > >>> wrote:
>> > >>>
>> > >>> > Hi Jun,
>> > >>> >
>> > >>> > 12. Sorry, I had to revert the change that removed `
>> > >>> > ClientQuotaCallback.quotaLimit()`. We are allowing quota
>> callbacks
>> > to
>> > >>> use
>> > >>> > custom metric tags. For each request, quota manager uses `
>> > >>> > ClientQuotaCallback.quota()` to map (user-principal, client-id) to
>> > the
>> > >>> > metric tags that determine which clients share the quota. When
>> quotas
>> > >>> are
>> > >>> > updated using  `updateQuota` or `updatePartitionMetadata`,
>> existing
>> > >>> metrics
>> > >>> > need to updated, but quota managers don't have a reverse mapping
>> of
>> > >>> metric
>> > >>> > tags to (user-principal, client-id) for
>> invoking`ClientQuotaCallback.
>> > >>> > quota()
>> > >>> > ` . Callbacks cannot return all updated metrics since they don't
>> have
>> > >>> > access to the metrics object and we don't want to require
>> callbacks
>> > to
>> > >>> > track all the entities for which metrics have been created (since
>> > they
>> > >>> may
>> > >>> > contain client-ids and hence need expiring). With the extra
>> method,
>> > >>> quota
>> > >>> > manager traverses the metric list after `updateQuota` or `
>> > >>> > updatePartitionMetadata` and obtains the latest value
>> corresponding
>> > to
>> > >>> each
>> > >>> > metric based on the tags using `ClientQuotaCallback.quotaLimi
>> t()`.
>> > >>> >
>> > >>> > An alternative may be to delay quota metrics updates until the
>> next
>> > >>> request
>> > >>> > that uses the metric. When we get sensors, we can check if the
>> quota
>> > >>> > configured in the metric matches the value returned by `
>> > >>> > ClientQuotaCallback.quota()`. This will be slightly more expensive
>> > >>> since we
>> > >>> > need to check on every request, but the callback API as well as
>> the
>> > >>> quota
>> > >>> > manager update code path would be simpler. What do you think?
>> > >>> >
>> > >>> > Thanks,
>> > >>> >
>> > >>> > Rajini
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > On Wed, Mar 14, 2018 at 11:21 PM, Rajini Sivaram <
>> > >>> rajinisiva...@gmail.com>
>> > >>> > wrote:
>> > >>> >
>> > >>> > > Hi Jun,
>> > >>> > >
>> > >>> > > Thank you for reviewing the KIP.
>> > >>> > >
>> > >>> 

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

[Rajini Sivaram]

Hi Ron,

For the password example, you could define a login CallbackHandler that
processes PasswordCallback to provide passwords. We don't currently do this
with PLAIN/SCRAM because login callback handlers were not configurable
earlier and we haven't updated the login modules to do this. But that could
be one way of providing passwords and integrating with other password
sources, now that we have configurable login callback handlers. I was
wondering whether similar approach could be used for the parameters that
OAuth needed to obtain at runtime. We could still have this KIP with
built-in substitutable types to handle common cases like getting options
from a file without writing any code. But I wasn't sure if there were OAuth
options that couldn't be handled as callbacks using the login callback
handler.

On Thu, Apr 5, 2018 at 10:25 PM, Ron Dagostino  wrote:

> Hi Rajini.  Thanks for the questions.  I could see someone wanting to
> retrieve a password from a vended password vault solution (for example);
> that is the kind of scenario that the ability to add new substitutable
> types would be meant for.  I do still consider this KIP 269 to be a
> prerequisite for the SASL/OAUTHBEARER KIP 255.  I am open to a different
> perspective in case I missed or misunderstood your point.
>
> Ron
>
> On Thu, Apr 5, 2018 at 8:13 AM, Rajini Sivaram 
> wrote:
>
> > Hi Ron,
> >
> > Now that login callback handlers are configurable, is this KIP still a
> > pre-req for OAuth? I was wondering whether we still need the ability to
> add
> > new substitutable types or whether it would be sufficient to add the
> > built-in ones to read from file etc.
> >
> >
> > On Thu, Mar 29, 2018 at 6:48 AM, Ron Dagostino 
> wrote:
> >
> > > Hi everyone.  There have been no comments on this KIP, so I intend to
> put
> > > it to a vote next week if there are no comments that might entail
> changes
> > > between now and then.  Please take a look in the meantime if you wish.
> > >
> > > Ron
> > >
> > > On Thu, Mar 15, 2018 at 2:36 PM, Ron Dagostino 
> > wrote:
> > >
> > > > Hi everyone.
> > > >
> > > > I created KIP-269: Substitution Within Configuration Values
> > > >  > > 269+Substitution+Within+Configuration+Values>
> > > > (https://cwiki.apache.org/confluence/display/KAFKA/KIP+269+
> > > > Substitution+Within+Configuration+Values
> > > >  > > action?pageId=75968876>
> > > > ).
> > > >
> > > > This KIP proposes adding support for substitution within client JAAS
> > > > configuration values for PLAIN and SCRAM-related SASL mechanisms in a
> > > > backwards-compatible manner and making the functionality available to
> > > other
> > > > existing (or future) configuration contexts where it is deemed
> > > appropriate.
> > > >
> > > > This KIP was extracted from (and is now a prerequisite for) KIP-255:
> > > > OAuth Authentication via SASL/OAUTHBEARER
> > > >  > > action?pageId=75968876>
> > > > based on discussion of that KIP.
> > > >
> > > > Ron
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

[Matthias J. Sax]

Ewen,

I cannot completely follow your argument. Can you elaborate a little
bit? After reading you mail, I am not sure if you prefer config
inheritance or not? And if, to what extend?


> In that mode if you override *anything*, you
>> should specify *everything*.

Can you give an example for this?


> But if it was just, e.g. group.id, client.id,
>> and offset.reset that needed adjustment for a restoreConsumer, that would
>> be the default and everything else is inherited.

Don't understand this part.


> I think
>> inheritance in these configuration setups needs to be approached very
>> carefully.

Agreed. I think that our approach is carefully designed though.


For follow up on Guozhang's argument, I agree with him, that for most
users the existing prefix would be good enough and the three new
prefixes are for advanced users.



-Matthias

On 4/5/18 11:37 AM, Guozhang Wang wrote:
> Ewen, thanks for your thoughts.
> 
> Just to clarify the current KIP does not propose for four new prefixes, but
> three plus the existing one. So it is
> 
> "consumer"
> "main.consumer"
> "restore.consumer"
> "global.consumer"
> 
> 
> If we design the configs for the first time, I'd be in favor of only
> keeping the last three prefixes. But as of now we have a compatibility
> issue to consider. So the question is if it is worthwhile to break the
> compatibility and enforce users to make code changes. My rationale is that:
> 
> 1) for normal users they would not bother overriding configs for different
> types of consumers, where "consumer" prefix is good enough for them; and
> today they probably have already made those overrides via "consumer".
> 
> 2) for advanced users they would need some additional overrides for
> different types of consumers, and they would go ahead and learn about the
> other three prefixes and set them there.
> 
> 
> I agree that four prefixes would be more confusing, but if we think use
> case 1)'s popularity is much larger than use case 2), which by the way we
> can still debate on, then I'd argue it's better to not force normal user
> groups from 1) to make code changes to make advanced users from 2) less
> confused about the hierarchy.
> 
> 
> Guozhang
> 
> 
> 
> 
> 
> 
> On Wed, Apr 4, 2018 at 11:23 PM, Ewen Cheslack-Postava 
> wrote:
> 
>> I think this model is more confusing than it needs to be.
>>
>> We end up with 4 prefixes despite only have 3 types of consumers. We have
>> prefixes for: "base", "main", "global", and "restore". However, we only
>> instantiate consumers of type "main", "global", and "restore".
>>
>> Until now, we've only had two types of configs mapping to two types of
>> consumers, despite internally having some common shared configs as a
>> baseline to bootstrap the two "public" ones (see
>> StreamsConfig.getCommonConsumerConfigs). Do we want to complicate this to
>> 4
>> levels of "public" configs when there are only 3 types of concrete configs
>> we instantiate?
>>
>> More generally, I worry that we're optimizing too much to avoid copy/paste
>> in less common cases to the point that we would confuse users with yet more
>> concepts before they can even write their configuration. What if we took an
>> (perhaps modified) all or nothing approach to inheriting from the the
>> "main" consumer properties? In that mode if you override *anything*, you
>> should specify *everything*. But if it was just, e.g. group.id, client.id,
>> and offset.reset that needed adjustment for a restoreConsumer, that would
>> be the default and everything else is inherited. Same deal for a clearly
>> specified set of configs for global consumers that required override.
>>
>> I feel like I'm also concurrently seeing the opposite side of this problem
>> in Connect where we namespaced and didn't proactively implement
>> inheritance; and while people find the config duplication annoying (and
>> confusing!), we inevitably find cases where they need it. I think
>> inheritance in these configuration setups needs to be approached very
>> carefully. Admittedly, some of the challenges in Connect don't appear here
>> (e.g. conflicts in producer/consumer config naming, since this is a
>> Consumer-only KIP), but similar problems arise.
>>
>> -Ewen
>>
>> On Wed, Apr 4, 2018 at 10:56 PM, Boyang Chen  wrote:
>>
>>> Thanks Guozhang! I already updated the pull request and KIP to deprecate
>>> getConsumerConfigs() function. Do you think we could move to a voting
>> stage
>>> now?
>>>
>>>
>>> 
>>> From: Guozhang Wang 
>>> Sent: Thursday, April 5, 2018 9:52 AM
>>> To: dev@kafka.apache.org
>>> Subject: Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different
>>> consumers
>>>
>>> I agree that renaming the method in this case may not worth it. Let's
>> keep
>>> the existing function names.
>>>
>>> On Wed, Apr 4, 2018 at 6:06 PM, Matthias J. Sax 
>>> wrote:
>>>
 Thanks for updating the KIP.

 

Re: [VOTE] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

[Matthias J. Sax]

+1 (binding)


-Matthias

On 4/5/18 4:36 AM, Ted Yu wrote:
> +1
>  Original message From: Mickael Maison 
>  Date: 4/5/18  1:42 AM  (GMT-08:00) To: dev 
>  Subject: Re: [VOTE] KIP-211: Revise Expiration 
> Semantics of Consumer Group Offsets 
> +1 (non-binding)
> Thanks for the KIP!
> 
> On Thu, Apr 5, 2018 at 8:08 AM, Jason Gustafson  wrote:
>> +1 Thanks Vahid!
>>
>> On Wed, Mar 28, 2018 at 7:27 PM, James Cheng  wrote:
>>
>>> +1 (non-binding)
>>>
>>> Thanks for all the hard work on this, Vahid!
>>>
>>> -James
>>>
 On Mar 28, 2018, at 10:34 AM, Vahid S Hashemian <
>>> vahidhashem...@us.ibm.com> wrote:

 Hi all,

 As I believe the feedback and suggestions on this KIP have been addressed
 so far, I'd like to start a vote.
 The KIP can be found at
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets

 Thanks in advance for voting :)

 --Vahid

>>>
>>>



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-219 - Improve Quota Communication

[Jonghyun Lee]

Hi,

I have been implementing KIP-219. I discussed the interface changes with
Becket Qin and Dong Lin, and we decided to bump up the protocol version of
ApiVersionsRequest and ApiVersionsResponse only, instead of bumping up all
requests/responses that may be throttled, to indicate clients whether or
not brokers perform throttling before sending out responses. We think this
is sufficient given that network client exchanges
ApiVersionsRequest/Response with each broker when establishing connection
to it and thus it can detect the broker's throttling behavior just by
examining the ApiVersionsResponse version.

Please respond to this e-mail if you have any questions or concerns.

Thanks,
Jon Lee


On Thu, Apr 5, 2018 at 2:29 PM, Becket Qin  wrote:

>
>
> On Thu, Nov 16, 2017 at 3:49 PM, Becket Qin  wrote:
>
>> Thanks Rajini,
>>
>> I updated the KIP wiki to clarify the scope of the KIP. To summarize, the
>> current quota has a few caveats:
>> 1. The brokers are only throttling the NEXT request even if the current
>> request is already violating quota. So the broker will always process at
>> least one request from the client.
>> 2. The brokers are not able to know the client id until they fully read
>> the request out of the sockets even if that client is being throttled.
>> 3. The brokers are not communicating to the clients promptly, so the
>> clients have to blindly wait and sometimes times out unnecessarily.
>>
>> This KIP only tries to address 3 but not 1 and 2 because A) those two
>> issues are sort of orthogonal to 3 and B) the solution to 1 and 2 are much
>> more complicated and worth a separate discussion.
>>
>> I'll wait till tomorrow and start a voting thread if there are further
>> concerns raised about the KIP.
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>> On Thu, Nov 16, 2017 at 11:47 AM, Rajini Sivaram > > wrote:
>>
>>> Hi Becket,
>>>
>>> The current user quota doesn't solve the problem. But I was thinking that
>>> if we could ensure we don't read more from the network than the quota
>>> allows, we may be able to fix the issue in a different way (throttling
>>> all
>>> connections, each for a limited time prior to reading large requests).
>>> But
>>> it would be more complex (and even more messy for client-id quotas), so I
>>> can understand why the solution you proposed in the KIP makes sense for
>>> the
>>> scenario that you described.
>>>
>>> Regards,
>>>
>>> Rajini
>>>
>>> On Tue, Nov 14, 2017 at 11:30 PM, Becket Qin 
>>> wrote:
>>>
>>> > Hi Rajini,
>>> >
>>> > We are using SSL so we could use user quota. But I am not sure if that
>>> > would solve the problem. The key issue in our case is that each broker
>>> can
>>> > only handle ~300 MB/s of incoming bytes, but the MapReduce job is
>>> trying to
>>> > push 1-2 GB/s, unless we can throttle the clients to 300 MB/s, the
>>> broker
>>> > cannot sustain. In order to do that, we need to be able to throttle
>>> > requests for more than request timeout, potentially a few minutes. It
>>> seems
>>> > neither user quota nor limited throttle time can achieve this.
>>> >
>>> > Thanks,
>>> >
>>> > Jiangjie (Becket) Qin
>>> >
>>> > On Tue, Nov 14, 2017 at 7:44 AM, Rajini Sivaram <
>>> rajinisiva...@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi Becket,
>>> > >
>>> > > For the specific scenario that you described, would it be possible
>>> to use
>>> > > user quotas rather than client-id quotas? With user quotas, perhaps
>>> we
>>> > can
>>> > > throttle more easily before reading requests as well (as you
>>> mentioned,
>>> > the
>>> > > difficulty with client-id quota is that we have to read partial
>>> requests
>>> > > and handle client-ids at network layer making that a much bigger
>>> change).
>>> > > If your clients are using SASL/SSL, I was wondering whether a
>>> solution
>>> > that
>>> > > improves user quotas and limits throttle time would work for you.
>>> > >
>>> > > Regards,
>>> > >
>>> > > Rajini
>>> > >
>>> > >
>>> > >
>>> > > On Thu, Nov 9, 2017 at 12:59 AM, Becket Qin 
>>> > wrote:
>>> > >
>>> > > > Since we will bump up the wire request version, another option is
>>> for
>>> > > > clients that are sending old request versions the broker can just
>>> keep
>>> > > the
>>> > > > current behavior. For clients sending the new request versions, the
>>> > > broker
>>> > > > can respond then mute the channel as described in the KIP wiki. In
>>> this
>>> > > > case, muting the channel is mostly for protection purpose. A
>>> correctly
>>> > > > implemented client should back off for throttle time before
>>> sending the
>>> > > > next request. The downside is that the broker needs to keep both
>>> logic
>>> > > and
>>> > > > it seems not gaining much benefit. So personally I prefer to just
>>> mute
>>> > > the
>>> > > > channel. But I am open to different opinions.
>>> > > >
>>> > > > Thanks,
>>> > > >
>>> > > > Jiangjie (Becket) Qin
>>> > > >

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

[Ron Dagostino]

Hi Rajini.  Thanks for the questions.  I could see someone wanting to
retrieve a password from a vended password vault solution (for example);
that is the kind of scenario that the ability to add new substitutable
types would be meant for.  I do still consider this KIP 269 to be a
prerequisite for the SASL/OAUTHBEARER KIP 255.  I am open to a different
perspective in case I missed or misunderstood your point.

Ron

On Thu, Apr 5, 2018 at 8:13 AM, Rajini Sivaram 
wrote:

> Hi Ron,
>
> Now that login callback handlers are configurable, is this KIP still a
> pre-req for OAuth? I was wondering whether we still need the ability to add
> new substitutable types or whether it would be sufficient to add the
> built-in ones to read from file etc.
>
>
> On Thu, Mar 29, 2018 at 6:48 AM, Ron Dagostino  wrote:
>
> > Hi everyone.  There have been no comments on this KIP, so I intend to put
> > it to a vote next week if there are no comments that might entail changes
> > between now and then.  Please take a look in the meantime if you wish.
> >
> > Ron
> >
> > On Thu, Mar 15, 2018 at 2:36 PM, Ron Dagostino 
> wrote:
> >
> > > Hi everyone.
> > >
> > > I created KIP-269: Substitution Within Configuration Values
> > >  > 269+Substitution+Within+Configuration+Values>
> > > (https://cwiki.apache.org/confluence/display/KAFKA/KIP+269+
> > > Substitution+Within+Configuration+Values
> > >  > action?pageId=75968876>
> > > ).
> > >
> > > This KIP proposes adding support for substitution within client JAAS
> > > configuration values for PLAIN and SCRAM-related SASL mechanisms in a
> > > backwards-compatible manner and making the functionality available to
> > other
> > > existing (or future) configuration contexts where it is deemed
> > appropriate.
> > >
> > > This KIP was extracted from (and is now a prerequisite for) KIP-255:
> > > OAuth Authentication via SASL/OAUTHBEARER
> > >  > action?pageId=75968876>
> > > based on discussion of that KIP.
> > >
> > > Ron
> > >
> >
>

Build failed in Jenkins: kafka-1.1-jdk7 #112

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-6748: double check before scheduling a new task after the

--
[...truncated 414.03 KB...]

kafka.controller.ReplicaStateMachineTest > 
testNewReplicaToOfflineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testNewReplicaToOfflineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToOnlineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToOnlineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNonexistentReplicaToReplicaDeletionStartedTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNonexistentReplicaToReplicaDeletionStartedTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToReplicaDeletionSuccessfulTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToReplicaDeletionSuccessfulTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOfflineReplicaToNonexistentReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOfflineReplicaToNonexistentReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToReplicaDeletionIneligibleTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToReplicaDeletionIneligibleTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToReplicaDeletionStartedTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToReplicaDeletionStartedTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionSuccessfulTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionSuccessfulTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionIneligibleToReplicaDeletionStartedTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionIneligibleToReplicaDeletionStartedTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionStartedToOfflineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionStartedToOfflineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionStartedTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionStartedTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testOnlineReplicaToOnlineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testOnlineReplicaToOnlineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionIneligibleTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionIneligibleTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNonexistentReplicaToOfflineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNonexistentReplicaToOfflineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testReplicaDeletionStartedToReplicaDeletionSuccessfulTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testReplicaDeletionStartedToReplicaDeletionSuccessfulTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNonexistentReplicaToOnlineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNonexistentReplicaToOnlineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToNewReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToNewReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testReplicaDeletionStartedToReplicaDeletionIneligibleTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testReplicaDeletionStartedToReplicaDeletionIneligibleTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToOfflineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToOfflineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionIneligibleToNewReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionIneligibleToNewReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testReplicaDeletionIneligibleToOnlineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testReplicaDeletionIneligibleToOnlineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 

Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

[Jun Rao]

Hi, Ted, Vahid,

Added you to the invite.

Thanks,

Jun


On Thu, Apr 5, 2018 at 10:42 AM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Jun,
>
> I used to receive these invites, but didn't get this one.
> Please send me an invite. Thanks.
>
> Regards,
> --Vahid
>
>
>
> From:   Jun Rao 
> To: dev 
> Date:   04/05/2018 10:25 AM
> Subject:Kafka KIP meeting on Apr. 9 at 9:00am PDT
>
>
>
> Hi, Everyone,
>
> We plan to have a Kafka KIP meeting this coming Monday (Apr. 9)  at 9:00am
> PDT. If you plan to attend but haven't received an invite, please let me
> know. The following is the agenda.
>
> Agenda:
> KIP-253: Partition expansion
>
> Thanks,
>
> Jun
>
>
>
>
>

Re: [DISCUSS] KIP-279: Fix log divergence between leader and follower after fast leader fail over

[Ted Yu]

For the second alternative which was rejected (The follower sends all
sequences of {leader_epoch, end_offset})

bq. also increases the size of OffsetForLeaderEpoch request by at least
64bit

Though the size increases, the number of roundtrips is reduced meaningfully
which would increase the robustness of the solution.

Please expand the reasoning for unclean leader election for this
alternative.

Thanks

On Thu, Apr 5, 2018 at 12:17 PM, Anna Povzner  wrote:

> Hi,
>
>
> I just created KIP-279 to fix edge cases of log divergence for both clean
> and unclean leader election configs.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 279%3A+Fix+log+divergence+between+leader+and+follower+
> after+fast+leader+fail+over
>
>
> The KIP is basically a follow up to KIP-101, and proposes a slight
> extension to the replication protocol to fix edge cases where logs can
> diverge due to fast leader fail over.
>
>
> Feedback and suggestions are welcome!
>
>
> Thanks,
>
> Anna
>

Build failed in Jenkins: kafka-trunk-jdk9 #536

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-6748: double check before scheduling a new task after the

--
[...truncated 1.48 MB...]
kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateSavedOffsetWhenOffsetToClearToIsBetweenEpochs STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateSavedOffsetWhenOffsetToClearToIsBetweenEpochs PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica 
STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown PASSED

Jenkins build is back to normal : kafka-trunk-jdk9 #535

[Apache Jenkins Server]

See 

[DISCUSS] KIP-279: Fix log divergence between leader and follower after fast leader fail over

[Anna Povzner]

Hi,


I just created KIP-279 to fix edge cases of log divergence for both clean
and unclean leader election configs.


https://cwiki.apache.org/confluence/display/KAFKA/KIP-279%3A+Fix+log+divergence+between+leader+and+follower+after+fast+leader+fail+over


The KIP is basically a follow up to KIP-101, and proposes a slight
extension to the replication protocol to fix edge cases where logs can
diverge due to fast leader fail over.


Feedback and suggestions are welcome!


Thanks,

Anna

Jenkins build is back to normal : kafka-trunk-jdk8 #2529

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-281 ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User

[Alex Dunayevsky]

Sure, updated the table under 'Public Interfaces' by adding the TimeUnit
column.
Thank you

> In the table under 'Public Interfaces', please add a column with TimeUnit.
> Looks good overall.





‌

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

[Guozhang Wang]

Ewen, thanks for your thoughts.

Just to clarify the current KIP does not propose for four new prefixes, but
three plus the existing one. So it is

"consumer"
"main.consumer"
"restore.consumer"
"global.consumer"


If we design the configs for the first time, I'd be in favor of only
keeping the last three prefixes. But as of now we have a compatibility
issue to consider. So the question is if it is worthwhile to break the
compatibility and enforce users to make code changes. My rationale is that:

1) for normal users they would not bother overriding configs for different
types of consumers, where "consumer" prefix is good enough for them; and
today they probably have already made those overrides via "consumer".

2) for advanced users they would need some additional overrides for
different types of consumers, and they would go ahead and learn about the
other three prefixes and set them there.


I agree that four prefixes would be more confusing, but if we think use
case 1)'s popularity is much larger than use case 2), which by the way we
can still debate on, then I'd argue it's better to not force normal user
groups from 1) to make code changes to make advanced users from 2) less
confused about the hierarchy.


Guozhang






On Wed, Apr 4, 2018 at 11:23 PM, Ewen Cheslack-Postava 
wrote:

> I think this model is more confusing than it needs to be.
>
> We end up with 4 prefixes despite only have 3 types of consumers. We have
> prefixes for: "base", "main", "global", and "restore". However, we only
> instantiate consumers of type "main", "global", and "restore".
>
> Until now, we've only had two types of configs mapping to two types of
> consumers, despite internally having some common shared configs as a
> baseline to bootstrap the two "public" ones (see
> StreamsConfig.getCommonConsumerConfigs). Do we want to complicate this to
> 4
> levels of "public" configs when there are only 3 types of concrete configs
> we instantiate?
>
> More generally, I worry that we're optimizing too much to avoid copy/paste
> in less common cases to the point that we would confuse users with yet more
> concepts before they can even write their configuration. What if we took an
> (perhaps modified) all or nothing approach to inheriting from the the
> "main" consumer properties? In that mode if you override *anything*, you
> should specify *everything*. But if it was just, e.g. group.id, client.id,
> and offset.reset that needed adjustment for a restoreConsumer, that would
> be the default and everything else is inherited. Same deal for a clearly
> specified set of configs for global consumers that required override.
>
> I feel like I'm also concurrently seeing the opposite side of this problem
> in Connect where we namespaced and didn't proactively implement
> inheritance; and while people find the config duplication annoying (and
> confusing!), we inevitably find cases where they need it. I think
> inheritance in these configuration setups needs to be approached very
> carefully. Admittedly, some of the challenges in Connect don't appear here
> (e.g. conflicts in producer/consumer config naming, since this is a
> Consumer-only KIP), but similar problems arise.
>
> -Ewen
>
> On Wed, Apr 4, 2018 at 10:56 PM, Boyang Chen  wrote:
>
> > Thanks Guozhang! I already updated the pull request and KIP to deprecate
> > getConsumerConfigs() function. Do you think we could move to a voting
> stage
> > now?
> >
> >
> > 
> > From: Guozhang Wang 
> > Sent: Thursday, April 5, 2018 9:52 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different
> > consumers
> >
> > I agree that renaming the method in this case may not worth it. Let's
> keep
> > the existing function names.
> >
> > On Wed, Apr 4, 2018 at 6:06 PM, Matthias J. Sax 
> > wrote:
> >
> > > Thanks for updating the KIP.
> > >
> > > One more comment. Even if we don't expect users to call
> > > `StreamsConfig#getConsumerConfigs()` it is still public API. Thus, we
> > > cannot just rename the method.
> > >
> > > I think we have two options: either we keep the current name or we
> > > deprecate the method and introduce `getMainConsumerConfigs()` in
> > > parallel. The deprecated method would just call the new method.
> > >
> > > I am not sure if we gain a lot renaming the method, thus I have a
> slight
> > > preference in just keeping the method name as is. (But I am also ok to
> > > rename it, if people prefer this)
> > >
> > > -Matthias
> > >
> > >
> > > On 4/3/18 1:59 PM, Bill Bejeck wrote:
> > > > Boyang,
> > > >
> > > > Thanks for making changes to the KIP,  I'm +1 on the updated version.
> > > >
> > > > -Bill
> > > >
> > > > On Tue, Apr 3, 2018 at 1:14 AM, Boyang Chen 
> > wrote:
> > > >
> > > >> Hey friends,
> > > >>
> > > >>
> > > >> both KIP > > 

[jira] [Resolved] (KAFKA-6748) Scheduler cannot be cancelled from Punctuator

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-6748.
--
   Resolution: Fixed
Fix Version/s: (was: 1.0.2)
   1.2.0

> Scheduler cannot be cancelled from Punctuator
> -
>
> Key: KAFKA-6748
> URL: https://issues.apache.org/jira/browse/KAFKA-6748
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.1.0, 1.0.1
>Reporter: Frederic Arno
>Assignee: Frederic Arno
>Priority: Major
> Fix For: 1.2.0, 1.1.1
>
>
> A Scheduler cannot be cancelled from within the scheduled punctuator, I will 
> post a test case to illustrate the problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6413) ReassignPartitionsCommand#parsePartitionReassignmentData() should give better error message when JSON is malformed

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6413.
--
Resolution: Duplicate

Resolving as duplicate of KAFKA-6084.

> ReassignPartitionsCommand#parsePartitionReassignmentData() should give better 
> error message when JSON is malformed
> --
>
> Key: KAFKA-6413
> URL: https://issues.apache.org/jira/browse/KAFKA-6413
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ted Yu
>Priority: Minor
>  Labels: json
>
> In this thread: 
> http://search-hadoop.com/m/Kafka/uyzND1J9Hizcxo0X?subj=Partition+reassignment+data+file+is+empty
>  , Allen gave an example JSON string with extra comma where 
> partitionsToBeReassigned returned by 
> ReassignPartitionsCommand#parsePartitionReassignmentData() was empty.
> I tried the following example where a right bracket is removed:
> {code}
> val (partitionsToBeReassigned, replicaAssignment) = 
> ReassignPartitionsCommand.parsePartitionReassignmentData(
> 
> "{\"version\":1,\"partitions\":[{\"topic\":\"metrics\",\"partition\":0,\"replicas\":[1,2]},{\"topic\":\"metrics\",\"partition\":1,\"replicas\":[2,3]},}");
> {code}
> The returned partitionsToBeReassigned is empty (and no exception was thrown).
> The parser should give better error message for malformed JSON string.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

[Vahid S Hashemian]

Hi Jun,

I used to receive these invites, but didn't get this one.
Please send me an invite. Thanks.

Regards,
--Vahid



From:   Jun Rao 
To: dev 
Date:   04/05/2018 10:25 AM
Subject:Kafka KIP meeting on Apr. 9 at 9:00am PDT



Hi, Everyone,

We plan to have a Kafka KIP meeting this coming Monday (Apr. 9)  at 9:00am
PDT. If you plan to attend but haven't received an invite, please let me
know. The following is the agenda.

Agenda:
KIP-253: Partition expansion

Thanks,

Jun

Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

[Ted Yu]

Jun:
I would be interested.

Is there going to be an online meeting ?

Thanks

On Thu, Apr 5, 2018 at 10:24 AM, Jun Rao  wrote:

> Hi, Everyone,
>
> We plan to have a Kafka KIP meeting this coming Monday (Apr. 9)  at 9:00am
> PDT. If you plan to attend but haven't received an invite, please let me
> know. The following is the agenda.
>
> Agenda:
> KIP-253: Partition expansion
>
> Thanks,
>
> Jun
>

Kafka KIP meeting on Apr. 9 at 9:00am PDT

[Jun Rao]

Hi, Everyone,

We plan to have a Kafka KIP meeting this coming Monday (Apr. 9)  at 9:00am
PDT. If you plan to attend but haven't received an invite, please let me
know. The following is the agenda.

Agenda:
KIP-253: Partition expansion

Thanks,

Jun

Build failed in Jenkins: kafka-trunk-jdk9 #534

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] KAFKA-6741:  Disable Selector's idle connection timeout in

--
[...truncated 1.48 MB...]
kafka.message.ByteBufferMessageSetTest > 
testWriteToChannelThatConsumesPartially STARTED

kafka.message.ByteBufferMessageSetTest > 
testWriteToChannelThatConsumesPartially PASSED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent STARTED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.message.ByteBufferMessageSetTest > testWriteTo STARTED

kafka.message.ByteBufferMessageSetTest > testWriteTo PASSED

kafka.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.message.ByteBufferMessageSetTest > testIterator STARTED

kafka.message.ByteBufferMessageSetTest > testIterator PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testSessionExpireListenerMetrics STARTED

kafka.metrics.MetricsTest > testSessionExpireListenerMetrics PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testClusterIdMetric STARTED

kafka.metrics.MetricsTest > testClusterIdMetric PASSED

kafka.metrics.MetricsTest > testControllerMetrics STARTED

kafka.metrics.MetricsTest > testControllerMetrics PASSED

kafka.metrics.MetricsTest > testWindowsStyleTagNames STARTED

kafka.metrics.MetricsTest > testWindowsStyleTagNames PASSED

kafka.metrics.MetricsTest > testMetricsLeak STARTED

kafka.metrics.MetricsTest > testMetricsLeak PASSED

kafka.metrics.MetricsTest > testBrokerTopicMetricsBytesInOut STARTED

kafka.metrics.MetricsTest > testBrokerTopicMetricsBytesInOut PASSED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testPeriodicTokenExpiry STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testPeriodicTokenExpiry PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testTokenRequestsWithDelegationTokenDisabled STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testTokenRequestsWithDelegationTokenDisabled PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testDescribeToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testDescribeToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testCreateToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testCreateToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testExpireToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testExpireToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testRenewToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testRenewToken 
PASSED

kafka.security.auth.ResourceTypeTest > testJavaConversions STARTED

kafka.security.auth.ResourceTypeTest > testJavaConversions PASSED

kafka.security.auth.ResourceTypeTest > testFromString 

[DISCUSS] KIP-282: Add the listener name to the authentication context

[Mickael Maison]

Hi all,

I have submitted KIP-282 to add the listener name to the authentication context:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-282%3A+Add+the+listener+name+to+the+authentication+context

This is a very minor KIP to simplify identifying the source of a
connection instead of relying on parsing the client address when
building Principals.

Feedback and suggestions are welcome!

Thanks

Re: [VOTE] KIP-274: Kafka Streams Skipped Records Metrics

[John Roesler]

Thanks everyone,

Voting for KIP-274 has passed with 3 binding and 4 non-binding +1s (and no
objections).

Thanks for your time in considering this KIP,
-John

On Thu, Apr 5, 2018 at 4:48 AM, Damian Guy  wrote:

> +1
>
> On Thu, 5 Apr 2018 at 01:56 Matthias J. Sax  wrote:
>
> > +1 (binding) on the latest proposal.
> >
> >
> > -Matthias
> >
> >
> > On 4/2/18 12:10 PM, Richard Yu wrote:
> > > +1
> > >
> > > On Mon, Apr 2, 2018 at 8:42 AM, Guozhang Wang 
> > wrote:
> > >
> > >> +1 (binding).
> > >>
> > >> On Mon, Apr 2, 2018 at 7:22 AM, Ted Yu  wrote:
> > >>
> > >>> +1
> > >>>
> > >>> On Mon, Apr 2, 2018 at 7:11 AM, Bill Bejeck 
> wrote:
> > >>>
> >  Thanks for the KIP.
> > 
> >  +1
> > 
> >  -Bill
> > 
> >  On Mon, Apr 2, 2018 at 10:09 AM, John Roesler 
> > >> wrote:
> > 
> > > Dear Kafka community,
> > >
> > > I am proposing KIP-274 to improve visibility when Streams skips
> > >> invalid
> > > records.
> > >
> > > The proposal is to simplify the metrics and add warning logs.
> > >
> > > Please find details in the wiki:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 274%3A+Kafka+Streams+Skipped+Records+Metrics
> > >
> > > Thanks,
> > >
> > > -John
> > >
> > 
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> >
> >
>

Build failed in Jenkins: kafka-trunk-jdk8 #2528

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] KAFKA-6694: The Trogdor Coordinator should support filtering 
task

[rajinisivaram] KAFKA-6741:  Disable Selector's idle connection timeout in

--
[...truncated 423.23 KB...]

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetDataExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetDataExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testMixedPipeline STARTED

kafka.zookeeper.ZooKeeperClientTest > testMixedPipeline PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetDataExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetDataExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testDeleteExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testDeleteExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiry STARTED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiry PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetDataNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetDataNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testDeleteNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testDeleteNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testExistsExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testExistsExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDeletion STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDeletion PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetAclNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetAclNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testStateChangeHandlerForAuthFailure 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testStateChangeHandlerForAuthFailure 
PASSED

kafka.network.SocketServerTest > testGracefulClose STARTED

kafka.network.SocketServerTest > testGracefulClose PASSED

kafka.network.SocketServerTest > controlThrowable STARTED

kafka.network.SocketServerTest > controlThrowable PASSED

kafka.network.SocketServerTest > testRequestMetricsAfterStop STARTED

kafka.network.SocketServerTest > testRequestMetricsAfterStop PASSED

kafka.network.SocketServerTest > testConnectionIdReuse STARTED

kafka.network.SocketServerTest > testConnectionIdReuse PASSED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
STARTED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
PASSED

kafka.network.SocketServerTest > testProcessorMetricsTags STARTED

kafka.network.SocketServerTest > testProcessorMetricsTags PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > testConnectionId STARTED

kafka.network.SocketServerTest > testConnectionId PASSED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics STARTED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics PASSED

kafka.network.SocketServerTest > testNoOpAction STARTED

kafka.network.SocketServerTest > testNoOpAction PASSED

kafka.network.SocketServerTest > simpleRequest STARTED

kafka.network.SocketServerTest > simpleRequest PASSED

kafka.network.SocketServerTest > closingChannelException STARTED

kafka.network.SocketServerTest > closingChannelException PASSED

kafka.network.SocketServerTest > testIdleConnection STARTED

kafka.network.SocketServerTest > testIdleConnection PASSED

kafka.network.SocketServerTest > 
testClientDisconnectionWithStagedReceivesFullyProcessed STARTED

kafka.network.SocketServerTest > 
testClientDisconnectionWithStagedReceivesFullyProcessed PASSED

kafka.network.SocketServerTest > testZeroMaxConnectionsPerIp STARTED

kafka.network.SocketServerTest > testZeroMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > testMetricCollectionAfterShutdown STARTED

kafka.network.SocketServerTest > testMetricCollectionAfterShutdown PASSED

kafka.network.SocketServerTest > testSessionPrincipal STARTED

kafka.network.SocketServerTest > testSessionPrincipal PASSED

kafka.network.SocketServerTest > configureNewConnectionException STARTED

kafka.network.SocketServerTest > configureNewConnectionException PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides PASSED

kafka.network.SocketServerTest > processNewResponseException STARTED

[jira] [Created] (KAFKA-6751) Make max.connections.per.ip.overrides a dynamic config

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-6751:
--

 Summary: Make max.connections.per.ip.overrides a dynamic config
 Key: KAFKA-6751
 URL: https://issues.apache.org/jira/browse/KAFKA-6751
 Project: Kafka
  Issue Type: Improvement
Reporter: Jason Gustafson


It might be useful to be able to update this config dynamically since we 
occasionally run into situations where a particular host (or set of hosts) is 
causing some trouble for the broker.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Security Vulnerabilities

[Nikolaos Strongioglou]

Is there a list including Kafka's outstanding security vulnerability issues
like the ones posted in the majority of CVE databases. I am looking for
something like this -->
https://www.cvedetails.com/product/27453/Apache-Zookeeper.html?vendor_id=45

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

[Vahid S Hashemian]

Hi Matthias,

Thanks a lot for reviewing the KIP and the clarification on streams 
protocol type for consumer groups.
I have updated the KIP with your suggestion.
I'm assuming the cast vote so far will remain valid since this is a minor 
change.

Cheers,
--Vahid




From:   "Matthias J. Sax" 
To: dev@kafka.apache.org
Date:   04/04/2018 06:28 PM
Subject:Re: [DISCUSS] KIP-211: Revise Expiration Semantics of 
Consumer Group Offsets



I was just reading the whole KIP for the first time. Nice work!


One minor comment. In the table of the standalone consumer, the first
line, first column says:

> = Empty
> (protocolType = Some("consumer"))

I think this should be

> = Empty
> (protocolType != None)
Note, that for example KafkaStreams uses a different protocol type
(namely "stream"). Also, other consumer must implement their own
partition assignors, too, with other names.



-Matthias


On 3/26/18 1:44 PM, Vahid S Hashemian wrote:
> Hi all,
> 
> Thanks for the feedback on this KIP so far.
> 
> If there is no additional feedback, I'll start a vote on Wed.
> 
> Thanks.
> --Vahid
> 
> 

[attachment "signature.asc" deleted by Vahid S Hashemian/Silicon 
Valley/IBM] 

Re: [DISCUSS] KIP-280: Enhanced log compaction

[Ted Yu]

In the 'Proposed Changes' section, can you expand 'OCC' ?

bq. Specifically changing this to anything other than "*offset*"

Is it possible to enumerate the keys ? In the future, more metadata would
be defined in record header - it is better to avoid collision.

Cheers

On Thu, Apr 5, 2018 at 2:05 AM, Luís Cabral 
wrote:

>
> This is embarassingly hard to fix... going again...
> 
> KIP-280:  https://cwiki.apache.org/confluence/display/
> KAFKA/KIP-280%3A+Enhanced+log+compaction
> -
> Pull-4822:  https://github.com/apache/kafka/pull/4822
>
>
> On Thursday, April 5, 2018, 11:03:22 AM GMT+2, Luís Cabral
>  wrote:
>
>   Fixing the links:KIP-280:  https://cwiki.apache.org/confluence/display/
> KAFKA/KIP-280%3A+Enhanced+log+compactionPull-4822:  https://
> github.com/apache/kafka/pull/4822
>
>
> On 2018/04/0508:44:00, Luís Cabral  wrote:
> > Helloall,>
> > Starting adiscussion for this feature.>
> >KIP-280   :  https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 280%3A+Enhanced+log+compactionPull-4822:  https://github.com/apache/
> kafka/pull/4822>
>
> > KindRegards,Luís>
>
>

Re: [DISCUSS] KIP-281 ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User

[Ted Yu]

In the table under 'Public Interfaces', please add a column with TimeUnit.

Looks good overall.

On Thu, Apr 5, 2018 at 7:14 AM, Alex Dunayevsky 
wrote:

> Hello friends,
>
> I would like to start a discussion thread for KIP-281:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 281%3A+ConsumerPerformance%3A+Increase+Polling+Loop+Timeout+
> and+Make+It+Reachable+by+the+End+User
>
> JIRA: KAFKA-6743: ConsumerPerformance fails to consume all messages on
> topics with large number of partitions
> 
>
> PR: https://github.com/apache/kafka/pull/4818
>
> Thank you,
> Alex Dunayevsky
>

Jenkins build is back to normal : kafka-trunk-jdk8 #2527

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-6750) Add listener name to AuthenticationContext

[Mickael Maison (JIRA)]

Mickael Maison created KAFKA-6750:
-

 Summary: Add listener name to AuthenticationContext
 Key: KAFKA-6750
 URL: https://issues.apache.org/jira/browse/KAFKA-6750
 Project: Kafka
  Issue Type: Improvement
  Components: security
Reporter: Mickael Maison
Assignee: Mickael Maison


It will be nice to have the ListenerName available in AuthenticationContexts.

Since we can have multiple listeners using the same security protocol, this 
would to identify precisely the origin of the connection in custom 
PrincipalBuilders.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSS] KIP-281 ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User

[Alex Dunayevsky]

Hello friends,

I would like to start a discussion thread for KIP-281:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-281%3A+ConsumerPerformance%3A+Increase+Polling+Loop+Timeout+and+Make+It+Reachable+by+the+End+User

JIRA: KAFKA-6743: ConsumerPerformance fails to consume all messages on
topics with large number of partitions


PR: https://github.com/apache/kafka/pull/4818

Thank you,
Alex Dunayevsky

[jira] [Resolved] (KAFKA-6741) Transient test failure: SslTransportLayerTest.testNetworkThreadTimeRecorded

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-6741.
---
   Resolution: Fixed
 Assignee: Manikumar
Fix Version/s: 1.2.0

> Transient test failure: SslTransportLayerTest.testNetworkThreadTimeRecorded
> ---
>
> Key: KAFKA-6741
> URL: https://issues.apache.org/jira/browse/KAFKA-6741
> Project: Kafka
>  Issue Type: Bug
>Reporter: Manikumar
>Assignee: Manikumar
>Priority: Minor
> Fix For: 1.2.0
>
>
> debug logs:
> {code}
>  [2018-04-03 14:51:33,365] DEBUG Added sensor with name connections-closed: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,368] DEBUG Added sensor with name connections-created: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,368] DEBUG Added sensor with name 
> successful-authentication: (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,368] DEBUG Added sensor with name 
> failed-authentication: (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,368] DEBUG Added sensor with name bytes-sent-received: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,369] DEBUG Added sensor with name bytes-sent: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,370] DEBUG Added sensor with name bytes-received: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,370] DEBUG Added sensor with name select-time: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,371] DEBUG Added sensor with name io-time: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,379] DEBUG Removed sensor with name connections-closed: 
> (org.apache.kafka.common.metrics.Metrics:447)
>  [2018-04-03 14:51:33,379] DEBUG Removed sensor with name 
> connections-created: (org.apache.kafka.common.metrics.Metrics:447)
>  [2018-04-03 14:51:33,379] DEBUG Removed sensor with name 
> successful-authentication: (org.apache.kafka.common.metrics.Metrics:447)
>  [2018-04-03 14:51:33,380] DEBUG Removed sensor with name 
> failed-authentication: (org.apache.kafka.common.metrics.Metrics:447)
>  [2018-04-03 14:51:33,380] DEBUG Removed sensor with name 
> bytes-sent-received: (org.apache.kafka.common.metrics.Metrics:447)
>  [2018-04-03 14:51:33,380] DEBUG Removed sensor with name bytes-sent: 
> (org.apache.kafka.common.metrics.Metrics:447)
>  [2018-04-03 14:51:33,380] DEBUG Removed sensor with name bytes-received: 
> (org.apache.kafka.common.metrics.Metrics:447)
>  [2018-04-03 14:51:33,382] DEBUG Removed sensor with name select-time: 
> (org.apache.kafka.common.metrics.Metrics:447)
>  [2018-04-03 14:51:33,382] DEBUG Removed sensor with name io-time: 
> (org.apache.kafka.common.metrics.Metrics:447)
>  [2018-04-03 14:51:33,382] DEBUG Added sensor with name connections-closed: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,382] DEBUG Added sensor with name connections-created: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,383] DEBUG Added sensor with name 
> successful-authentication: (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,383] DEBUG Added sensor with name 
> failed-authentication: (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,383] DEBUG Added sensor with name bytes-sent-received: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,384] DEBUG Added sensor with name bytes-sent: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,384] DEBUG Added sensor with name bytes-received: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,384] DEBUG Added sensor with name select-time: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,384] DEBUG Added sensor with name io-time: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,444] DEBUG Added sensor with name connections-closed: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,445] DEBUG Added sensor with name connections-created: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,445] DEBUG Added sensor with name 
> successful-authentication: (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,445] DEBUG Added sensor with name 
> failed-authentication: (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,446] DEBUG Added sensor with name bytes-sent-received: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,446] DEBUG Added sensor with name bytes-sent: 
> (org.apache.kafka.common.metrics.Metrics:414)
>  [2018-04-03 14:51:33,446] DEBUG Added sensor with name bytes-received: 
> 

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

[Rajini Sivaram]

Hi Ron,

Now that login callback handlers are configurable, is this KIP still a
pre-req for OAuth? I was wondering whether we still need the ability to add
new substitutable types or whether it would be sufficient to add the
built-in ones to read from file etc.


On Thu, Mar 29, 2018 at 6:48 AM, Ron Dagostino  wrote:

> Hi everyone.  There have been no comments on this KIP, so I intend to put
> it to a vote next week if there are no comments that might entail changes
> between now and then.  Please take a look in the meantime if you wish.
>
> Ron
>
> On Thu, Mar 15, 2018 at 2:36 PM, Ron Dagostino  wrote:
>
> > Hi everyone.
> >
> > I created KIP-269: Substitution Within Configuration Values
> >  269+Substitution+Within+Configuration+Values>
> > (https://cwiki.apache.org/confluence/display/KAFKA/KIP+269+
> > Substitution+Within+Configuration+Values
> >  action?pageId=75968876>
> > ).
> >
> > This KIP proposes adding support for substitution within client JAAS
> > configuration values for PLAIN and SCRAM-related SASL mechanisms in a
> > backwards-compatible manner and making the functionality available to
> other
> > existing (or future) configuration contexts where it is deemed
> appropriate.
> >
> > This KIP was extracted from (and is now a prerequisite for) KIP-255:
> > OAuth Authentication via SASL/OAUTHBEARER
> >  action?pageId=75968876>
> > based on discussion of that KIP.
> >
> > Ron
> >
>

Re: [DISCUSS] KIP-257 - Configurable Quota Management

[Rajini Sivaram]

Thanks, Ismael.

I have updated the KIP and the PR.

On Wed, Apr 4, 2018 at 7:37 PM, Ismael Juma  wrote:

> Sounds good to me Rajini. Good catch spotting this before it's included in
> a release. :)
>
> Ismael
>
> On Wed, Apr 4, 2018 at 11:13 AM, Rajini Sivaram 
> wrote:
>
> > For compatibility reasons, we are now using Java rather than Scala for
> all
> > pluggable interfaces including those on the broker. There is already a
> KIP
> > to move Authorizer to Java as well. As we will be removing support for
> Java
> > 7 in the next release, we can also use default methods in Java when we
> need
> > to update pluggable Java interfaces. So the plan is to use Java for all
> new
> > pluggable interfaces.
> >
> > We already have the package org.apache.kafka.server, under which we have
> > the sub-package for policies, so it makes sense to define quota callback
> as
> > a Java interface here too.
> >
> > If there are any concerns, please let me know. Otherwise I will update
> the
> > KIP and the associated PR.
> >
> > Thank you,
> >
> > Rajini
> >
> > On Thu, Mar 22, 2018 at 9:52 PM, Rajini Sivaram  >
> > wrote:
> >
> > > Since there all the comments so far have been addressed, I will start
> > vote
> > > for this KIP.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Thu, Mar 15, 2018 at 6:37 PM, Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > >> Thanks, Jun.
> > >>
> > >> 11. updatePartitionMetadata() provides all partitions with their
> leaders
> > >> so that callbacks that scale down quotas based on fraction of
> partitions
> > >> hosted on each broker may compute the scaling factor. Callbacks that
> > scale
> > >> up quotas based on the partition count hosted on each broker can
> simply
> > >> filter out the others. I have updated the Javadoc in the KIP.
> > >>
> > >> On Thu, Mar 15, 2018 at 5:24 PM, Jun Rao  wrote:
> > >>
> > >>> Hi, Rajini,
> > >>>
> > >>> Thanks for the explanation. It makes sense.
> > >>>
> > >>> 11. We probably want to clarify in the interface that every time when
> > >>> updatePartitionMetadata() is called, the full list of partitions
> whose
> > >>> leader is on this broker will be passed in?
> > >>>
> > >>> Jun
> > >>>
> > >>> On Thu, Mar 15, 2018 at 4:42 AM, Rajini Sivaram <
> > rajinisiva...@gmail.com
> > >>> >
> > >>> wrote:
> > >>>
> > >>> > Hi Jun,
> > >>> >
> > >>> > 12. Sorry, I had to revert the change that removed `
> > >>> > ClientQuotaCallback.quotaLimit()`. We are allowing quota callbacks
> > to
> > >>> use
> > >>> > custom metric tags. For each request, quota manager uses `
> > >>> > ClientQuotaCallback.quota()` to map (user-principal, client-id) to
> > the
> > >>> > metric tags that determine which clients share the quota. When
> quotas
> > >>> are
> > >>> > updated using  `updateQuota` or `updatePartitionMetadata`, existing
> > >>> metrics
> > >>> > need to updated, but quota managers don't have a reverse mapping of
> > >>> metric
> > >>> > tags to (user-principal, client-id) for
> invoking`ClientQuotaCallback.
> > >>> > quota()
> > >>> > ` . Callbacks cannot return all updated metrics since they don't
> have
> > >>> > access to the metrics object and we don't want to require callbacks
> > to
> > >>> > track all the entities for which metrics have been created (since
> > they
> > >>> may
> > >>> > contain client-ids and hence need expiring). With the extra method,
> > >>> quota
> > >>> > manager traverses the metric list after `updateQuota` or `
> > >>> > updatePartitionMetadata` and obtains the latest value corresponding
> > to
> > >>> each
> > >>> > metric based on the tags using `ClientQuotaCallback.quotaLimit()`.
> > >>> >
> > >>> > An alternative may be to delay quota metrics updates until the next
> > >>> request
> > >>> > that uses the metric. When we get sensors, we can check if the
> quota
> > >>> > configured in the metric matches the value returned by `
> > >>> > ClientQuotaCallback.quota()`. This will be slightly more expensive
> > >>> since we
> > >>> > need to check on every request, but the callback API as well as the
> > >>> quota
> > >>> > manager update code path would be simpler. What do you think?
> > >>> >
> > >>> > Thanks,
> > >>> >
> > >>> > Rajini
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Wed, Mar 14, 2018 at 11:21 PM, Rajini Sivaram <
> > >>> rajinisiva...@gmail.com>
> > >>> > wrote:
> > >>> >
> > >>> > > Hi Jun,
> > >>> > >
> > >>> > > Thank you for reviewing the KIP.
> > >>> > >
> > >>> > > 10. This is the current behaviour (this KIP retains the same
> > >>> behaviour
> > >>> > for
> > >>> > > the default quota callback). We include 'user' and 'client-id'
> tags
> > >>> in
> > >>> > > all the quota metrics, rather than omit tags at the moment.
> > >>> > >
> > >>> > > 11. Ah, I hadn't realised that. I wasn't expecting to include
> > deleted
> > >>> > > partitions in updatePartitionMetadata. I have updated the Javadoc
> > in
> 

Re: [VOTE] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

[Ted Yu]

+1
 Original message From: Mickael Maison 
 Date: 4/5/18  1:42 AM  (GMT-08:00) To: dev 
 Subject: Re: [VOTE] KIP-211: Revise Expiration Semantics 
of Consumer Group Offsets 
+1 (non-binding)
Thanks for the KIP!

On Thu, Apr 5, 2018 at 8:08 AM, Jason Gustafson  wrote:
> +1 Thanks Vahid!
>
> On Wed, Mar 28, 2018 at 7:27 PM, James Cheng  wrote:
>
>> +1 (non-binding)
>>
>> Thanks for all the hard work on this, Vahid!
>>
>> -James
>>
>> > On Mar 28, 2018, at 10:34 AM, Vahid S Hashemian <
>> vahidhashem...@us.ibm.com> wrote:
>> >
>> > Hi all,
>> >
>> > As I believe the feedback and suggestions on this KIP have been addressed
>> > so far, I'd like to start a vote.
>> > The KIP can be found at
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets
>> >
>> > Thanks in advance for voting :)
>> >
>> > --Vahid
>> >
>>
>>

Build failed in Jenkins: kafka-trunk-jdk8 #2526

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Don’t send the DeleteTopicsRequest for invalid topic names

--
[...truncated 3.53 MB...]
kafka.zookeeper.ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChange 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetDataExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetDataExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testMixedPipeline STARTED

kafka.zookeeper.ZooKeeperClientTest > testMixedPipeline PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetDataExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetDataExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testDeleteExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testDeleteExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiry STARTED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiry PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetDataNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetDataNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testDeleteNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testDeleteNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testExistsExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testExistsExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDeletion STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDeletion PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetAclNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetAclNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testStateChangeHandlerForAuthFailure 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testStateChangeHandlerForAuthFailure 
PASSED

kafka.network.SocketServerTest > testGracefulClose STARTED

kafka.network.SocketServerTest > testGracefulClose PASSED

kafka.network.SocketServerTest > controlThrowable STARTED

kafka.network.SocketServerTest > controlThrowable PASSED

kafka.network.SocketServerTest > testRequestMetricsAfterStop STARTED

kafka.network.SocketServerTest > testRequestMetricsAfterStop PASSED

kafka.network.SocketServerTest > testConnectionIdReuse STARTED

kafka.network.SocketServerTest > testConnectionIdReuse PASSED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
STARTED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
PASSED

kafka.network.SocketServerTest > testProcessorMetricsTags STARTED

kafka.network.SocketServerTest > testProcessorMetricsTags PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > testConnectionId STARTED

kafka.network.SocketServerTest > testConnectionId PASSED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics STARTED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics PASSED

kafka.network.SocketServerTest > testNoOpAction STARTED

kafka.network.SocketServerTest > testNoOpAction PASSED

kafka.network.SocketServerTest > simpleRequest STARTED

kafka.network.SocketServerTest > simpleRequest PASSED

kafka.network.SocketServerTest > closingChannelException STARTED

kafka.network.SocketServerTest > closingChannelException PASSED

kafka.network.SocketServerTest > testIdleConnection STARTED

kafka.network.SocketServerTest > testIdleConnection PASSED

kafka.network.SocketServerTest > 
testClientDisconnectionWithStagedReceivesFullyProcessed STARTED

kafka.network.SocketServerTest > 
testClientDisconnectionWithStagedReceivesFullyProcessed PASSED

kafka.network.SocketServerTest > testZeroMaxConnectionsPerIp STARTED

kafka.network.SocketServerTest > testZeroMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > testMetricCollectionAfterShutdown STARTED

kafka.network.SocketServerTest > testMetricCollectionAfterShutdown PASSED

kafka.network.SocketServerTest > testSessionPrincipal STARTED

kafka.network.SocketServerTest > testSessionPrincipal PASSED

kafka.network.SocketServerTest > configureNewConnectionException STARTED

kafka.network.SocketServerTest > configureNewConnectionException PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides PASSED

kafka.network.SocketServerTest > processNewResponseException STARTED

[jira] [Created] (KAFKA-6749) TopologyTestDriver fails when topoloy under test uses EXACTLY_ONCE

[Frederic Arno (JIRA)]

Frederic Arno created KAFKA-6749:


 Summary: TopologyTestDriver fails when topoloy under test uses 
EXACTLY_ONCE
 Key: KAFKA-6749
 URL: https://issues.apache.org/jira/browse/KAFKA-6749
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.1.0
Reporter: Frederic Arno


Stream processing topologies which are configured to use `EXACTLY_ONCE` 
processing guarantee cannot be tested with the `TopologyTestDriver`. Tests 
usually crash with `java.lang.IllegalStateException: MockProducer hasn't been 
initialized for transactions` within the second call to 
`TopologyTestDriver.pipeInput()`, the first call works fine.

Changing the processing guarantee to `AT_LEAST_ONCE` makes tests pass.

This is a problem because it is expected that proper processor topologies can 
be successfully tested using `TopologyTestDriver`, however `TopologyTestDriver` 
can't handle `EXACTLY_ONCE` and crashes during tests. To a developer, this 
usually means that there is something wrong with their processor topologies.

Kafka developpers can reproduce this by adding:
{code:java}
put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, 
StreamsConfig.EXACTLY_ONCE);{code}

 to line 88 of TopologyTestDriverTest: 
streams/test-utils/src/test/java/org/apache/kafka/streams/TopologyTestDriverTest.java

Originally reported here: 
[http://mail-archives.apache.org/mod_mbox/kafka-users/201804.mbox/%3C54ab29ad-44e1-35bd-9c16-c1d8d68a88db%40gmail.com%3E]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-274: Kafka Streams Skipped Records Metrics

[Damian Guy]

+1

On Thu, 5 Apr 2018 at 01:56 Matthias J. Sax  wrote:

> +1 (binding) on the latest proposal.
>
>
> -Matthias
>
>
> On 4/2/18 12:10 PM, Richard Yu wrote:
> > +1
> >
> > On Mon, Apr 2, 2018 at 8:42 AM, Guozhang Wang 
> wrote:
> >
> >> +1 (binding).
> >>
> >> On Mon, Apr 2, 2018 at 7:22 AM, Ted Yu  wrote:
> >>
> >>> +1
> >>>
> >>> On Mon, Apr 2, 2018 at 7:11 AM, Bill Bejeck  wrote:
> >>>
>  Thanks for the KIP.
> 
>  +1
> 
>  -Bill
> 
>  On Mon, Apr 2, 2018 at 10:09 AM, John Roesler 
> >> wrote:
> 
> > Dear Kafka community,
> >
> > I am proposing KIP-274 to improve visibility when Streams skips
> >> invalid
> > records.
> >
> > The proposal is to simplify the metrics and add warning logs.
> >
> > Please find details in the wiki:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 274%3A+Kafka+Streams+Skipped+Records+Metrics
> >
> > Thanks,
> >
> > -John
> >
> 
> >>>
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>
>

[jira] [Created] (KAFKA-6748) Scheduler cannot be cancelled from Punctuator

[Frederic Arno (JIRA)]

Frederic Arno created KAFKA-6748:


 Summary: Scheduler cannot be cancelled from Punctuator
 Key: KAFKA-6748
 URL: https://issues.apache.org/jira/browse/KAFKA-6748
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.1, 1.1.0
Reporter: Frederic Arno
Assignee: Frederic Arno
 Fix For: 1.0.2, 1.1.1


A Scheduler cannot be cancelled from within the scheduled punctuator, I will 
post a test case to illustrate the problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

[Jan Filipiak]

Yes I can be there.

On 04.04.2018 22:34, Jun Rao wrote:

Hi, Jan, Dong, John, Guozhang,

Perhaps it will be useful to have a KIP meeting to discuss this together as
a group. Would Apr. 9 (Monday) at 9:00am PDT work? If so, I will send out
an invite to the mailing list.

Thanks,

Jun


On Wed, Apr 4, 2018 at 1:25 AM, Jan Filipiak 
wrote:


Want to quickly step in here again because it is going places again.

The last part of the discussion is just a pain to read and completely
diverged from what I suggested without making the reasons clear to me.

I don't know why this happens here are my comments anyway.

@Guozhang: That Streams is working on automatic creating
copartition-usuable topics: great for streams, has literally nothing todo
with the KIP as we want to grow the
input topic. Everyone can reshuffle rel. easily but that is not what we
need todo, we need to grow the topic in question. After streams
automatically reshuffled the input topic
still has the same size and it didn't help a bit. I fail to see why this
is relevant. What am i missing here?

@Dong
I am still on the position that the current proposal brings us into the
wrong direction. Especially introducing PartitionKeyRebalanceListener
 From this point we can never move away to proper state full handling
without completely deprecating this creature from hell again.
Linear hashing is not the optimising step we have todo here. An interface
that when a topic is a topic its always the same even after it had
grown or shrunk is important. So from my POV I have major concerns that
this KIP is benefitial in its current state.

What is it that makes everyone so addicted to the idea of linear hashing?
not attractive at all for me.
And with statefull consumers still a complete mess. Why not stick with the
Kappa architecture???





On 03.04.2018 17:38, Dong Lin wrote:


Hey John,

Thanks much for your comments!!

I have yet to go through the emails of John/Jun/Guozhang in detail. But
let
me present my idea for how to minimize the delay for state loading for
stream use-case.

For ease of understanding, let's assume that the initial partition number
of input topics and change log topic are both 10. And initial number of
stream processor is also 10. If we only increase initial partition number
of input topics to 15 without changing number of stream processor, the
current KIP already guarantees in-order delivery and no state needs to be
moved between consumers for stream use-case. Next, let's say we want to
increase the number of processor to expand the processing capacity for
stream use-case. This requires us to move state between processors which
will take time. Our goal is to minimize the impact (i.e. delay) for
processing while we increase the number of processors.

Note that stream processor generally includes both consumer and producer.
In addition to consume from the input topic, consumer may also need to
consume from change log topic on startup for recovery. And producer may
produce state to the change log topic.



The solution will include the following steps:

1) Increase partition number of the input topic from 10 to 15. Since the
messages with the same key will still go to the same consume before and
after the partition expansion, this step can be done without having to
move
state between processors.

2) Increase partition number of the change log topic from 10 to 15. Note
that this step can also be done without impacting existing workflow. After
we increase partition number of the change log topic, key space may split
and some key will be produced to the newly-added partition. But the same
key will still go to the same processor (i.e. consumer) before and after
the partition. Thus this step can also be done without having to move
state
between processors.

3) Now, let's add 5 new consumers whose groupId is different from the
existing processor's groupId. Thus these new consumers will not impact
existing workflow. Each of these new consumers should consume two
partitions from the earliest offset, where these two partitions are the
same partitions that will be consumed if the consumers have the same
groupId as the existing processor's groupId. For example, the first of the
five consumers will consume partition 0 and partition 10. The purpose of
these consumers is to rebuild the state (e.g. RocksDB) for the processors
in advance. Also note that, by design of the current KIP, each consume
will
consume the existing partition of the change log topic up to the offset
before the partition expansion. Then they will only need to consume the
state of the new partition of the change log topic.

4) After consumers have caught up in step 3), we should stop these
consumers and add 5 new processors to the stream processing job. These 5
new processors should run in the same location as the previous 5 consumers
to re-use the state (e.g. RocksDB). And these processors' consumers should
consume partitions of the change log topic from 

Re: [DISCUSS] KIP-280: Enhanced log compaction

[Luís Cabral]

 
This is embarassingly hard to fix... going again...

KIP-280:  
https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction
-
Pull-4822:  https://github.com/apache/kafka/pull/4822


On Thursday, April 5, 2018, 11:03:22 AM GMT+2, Luís Cabral 
 wrote:  
 
  Fixing the links:KIP-280:  
https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compactionPull-4822:
  https://github.com/apache/kafka/pull/4822


On 2018/04/0508:44:00, Luís Cabral  wrote: 
> Helloall,> 
> Starting adiscussion for this feature.> 
>KIP-280   :  
>https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compactionPull-4822:
>  https://github.com/apache/kafka/pull/4822>

> KindRegards,Luís> 
  

Re: [DISCUSS] KIP-280: Enhanced log compaction

[Luís Cabral]

 Fixing the links:KIP-280:  
https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compactionPull-4822:
  https://github.com/apache/kafka/pull/4822


On 2018/04/0508:44:00, Luís Cabral  wrote: 
> Helloall,> 
> Starting adiscussion for this feature.> 
>KIP-280   :  
>https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compactionPull-4822:
>  https://github.com/apache/kafka/pull/4822>

> KindRegards,Luís> 

[jira] [Resolved] (KAFKA-4292) KIP-86: Configurable SASL callback handlers

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-4292.
---
   Resolution: Fixed
 Reviewer: Jun Rao
Fix Version/s: 1.2.0

> KIP-86: Configurable SASL callback handlers
> ---
>
> Key: KAFKA-4292
> URL: https://issues.apache.org/jira/browse/KAFKA-4292
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.10.1.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 1.2.0
>
>
> Implementation of KIP-86: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-86%3A+Configurable+SASL+callback+handlers



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSS] KIP-280: Enhanced log compaction

[Luís Cabral]

Hello all,
Starting a discussion for this feature.
KIP-280   :  
https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compactionPull-4822
 :  https://github.com/apache/kafka/pull/4822
Kind Regards,Luís

Re: [VOTE] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

[Mickael Maison]

+1 (non-binding)
Thanks for the KIP!

On Thu, Apr 5, 2018 at 8:08 AM, Jason Gustafson  wrote:
> +1 Thanks Vahid!
>
> On Wed, Mar 28, 2018 at 7:27 PM, James Cheng  wrote:
>
>> +1 (non-binding)
>>
>> Thanks for all the hard work on this, Vahid!
>>
>> -James
>>
>> > On Mar 28, 2018, at 10:34 AM, Vahid S Hashemian <
>> vahidhashem...@us.ibm.com> wrote:
>> >
>> > Hi all,
>> >
>> > As I believe the feedback and suggestions on this KIP have been addressed
>> > so far, I'd like to start a vote.
>> > The KIP can be found at
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets
>> >
>> > Thanks in advance for voting :)
>> >
>> > --Vahid
>> >
>>
>>

Re: [VOTE] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

[Jason Gustafson]

+1 Thanks Vahid!

On Wed, Mar 28, 2018 at 7:27 PM, James Cheng  wrote:

> +1 (non-binding)
>
> Thanks for all the hard work on this, Vahid!
>
> -James
>
> > On Mar 28, 2018, at 10:34 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
> >
> > Hi all,
> >
> > As I believe the feedback and suggestions on this KIP have been addressed
> > so far, I'd like to start a vote.
> > The KIP can be found at
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets
> >
> > Thanks in advance for voting :)
> >
> > --Vahid
> >
>
>

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

[Ewen Cheslack-Postava]

I think this model is more confusing than it needs to be.

We end up with 4 prefixes despite only have 3 types of consumers. We have
prefixes for: "base", "main", "global", and "restore". However, we only
instantiate consumers of type "main", "global", and "restore".

Until now, we've only had two types of configs mapping to two types of
consumers, despite internally having some common shared configs as a
baseline to bootstrap the two "public" ones (see
StreamsConfig.getCommonConsumerConfigs). Do we want to complicate this to 4
levels of "public" configs when there are only 3 types of concrete configs
we instantiate?

More generally, I worry that we're optimizing too much to avoid copy/paste
in less common cases to the point that we would confuse users with yet more
concepts before they can even write their configuration. What if we took an
(perhaps modified) all or nothing approach to inheriting from the the
"main" consumer properties? In that mode if you override *anything*, you
should specify *everything*. But if it was just, e.g. group.id, client.id,
and offset.reset that needed adjustment for a restoreConsumer, that would
be the default and everything else is inherited. Same deal for a clearly
specified set of configs for global consumers that required override.

I feel like I'm also concurrently seeing the opposite side of this problem
in Connect where we namespaced and didn't proactively implement
inheritance; and while people find the config duplication annoying (and
confusing!), we inevitably find cases where they need it. I think
inheritance in these configuration setups needs to be approached very
carefully. Admittedly, some of the challenges in Connect don't appear here
(e.g. conflicts in producer/consumer config naming, since this is a
Consumer-only KIP), but similar problems arise.

-Ewen

On Wed, Apr 4, 2018 at 10:56 PM, Boyang Chen  wrote:

> Thanks Guozhang! I already updated the pull request and KIP to deprecate
> getConsumerConfigs() function. Do you think we could move to a voting stage
> now?
>
>
> 
> From: Guozhang Wang 
> Sent: Thursday, April 5, 2018 9:52 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different
> consumers
>
> I agree that renaming the method in this case may not worth it. Let's keep
> the existing function names.
>
> On Wed, Apr 4, 2018 at 6:06 PM, Matthias J. Sax 
> wrote:
>
> > Thanks for updating the KIP.
> >
> > One more comment. Even if we don't expect users to call
> > `StreamsConfig#getConsumerConfigs()` it is still public API. Thus, we
> > cannot just rename the method.
> >
> > I think we have two options: either we keep the current name or we
> > deprecate the method and introduce `getMainConsumerConfigs()` in
> > parallel. The deprecated method would just call the new method.
> >
> > I am not sure if we gain a lot renaming the method, thus I have a slight
> > preference in just keeping the method name as is. (But I am also ok to
> > rename it, if people prefer this)
> >
> > -Matthias
> >
> >
> > On 4/3/18 1:59 PM, Bill Bejeck wrote:
> > > Boyang,
> > >
> > > Thanks for making changes to the KIP,  I'm +1 on the updated version.
> > >
> > > -Bill
> > >
> > > On Tue, Apr 3, 2018 at 1:14 AM, Boyang Chen 
> wrote:
> > >
> > >> Hey friends,
> > >>
> > >>
> > >> both KIP > >> 276+Add+StreamsConfig+prefix+for+different+consumers> and pull
> request<
> > >> https://github.com/apache/kafka/pull/4805> are updated. Feel free to
> > take
> > >> another look.
> > >>
> > >>
> > >>
> > >> Thanks for your valuable feedback!
> > >>
> > >>
> > >> Best,
> > >>
> > >> Boyang
> > >>
> > >> 
> > >> From: Boyang Chen 
> > >> Sent: Tuesday, April 3, 2018 11:39 AM
> > >> To: dev@kafka.apache.org
> > >> Subject: Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different
> > >> consumers
> > >>
> > >> Thanks Matthias, Ted and Guozhang for the inputs. I shall address them
> > in
> > >> next round.
> > >>
> > >>
> > >> 
> > >> From: Matthias J. Sax 
> > >> Sent: Tuesday, April 3, 2018 4:43 AM
> > >> To: dev@kafka.apache.org
> > >> Subject: Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different
> > >> consumers
> > >>
> > >> Yes, your examples make sense to me. That was the idea behind the
> > proposal.
> > >>
> > >>
> > >> -Matthias
> > >>
> > >> On 4/2/18 11:25 AM, Guozhang Wang wrote:
> > >>> @Matthias
> > >>>
> > >>> That's a good question: I think adding another config for the main
> > >> consumer
> > >>> makes good tradeoffs between compatibility and new user convenience.
> > Just
> > >>> to clarify, from user's pov on upgrade:
> > >>>
> > >>> 1) I'm already overriding some consumer configs, and now I want to
> > >> override
> > >>> these values