[jira] [Resolved] (KAFKA-5640) Look into making acks=all the default setting

[Apurva Mehta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apurva Mehta resolved KAFKA-5640.
-
Resolution: Duplicate

This is a dup of https://issues.apache.org/jira/browse/KAFKA-5796

> Look into making acks=all the default setting
> -
>
> Key: KAFKA-5640
> URL: https://issues.apache.org/jira/browse/KAFKA-5640
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
> Fix For: 1.0.0
>
>
> KAFKA-5494 proposed dropping the requirement for 
> {{max.inflight.requests.per.connection=1}} for the idempotent producer. 
> That is a stepping stone to enabling the idempotent producer by default 
> without sacrificing performance.
> A further step would be making {{acks=all}} the default setting as well. 
> Then, with {{enable.idempotence=true}}, 
> {{max.inflight.requests.per.connection=5}}, {{acks=all}}, 
> {{retries=MAX_INT}}, we would have exactly once semantics with strong 
> durability guarantees. 
> This particular ticket is about investigating the performance degradation 
> caused by {{acks=all}}. How much does throughput degrade? If it is 
> significant, are there low hanging fruit in terms of code or config changes 
> which would allow us to bridge most of the gap?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5796) Understand performance implications of acks=all and potential ways to reduce it

[Apurva Mehta (JIRA)]

Apurva Mehta created KAFKA-5796:
---

 Summary: Understand performance implications of acks=all and 
potential ways to reduce it
 Key: KAFKA-5796
 URL: https://issues.apache.org/jira/browse/KAFKA-5796
 Project: Kafka
  Issue Type: Sub-task
 Environment: To get exactly once semantics, we need acks=all. However, 
we know that there is a latency and throughput impact with acks=all when 
compared with acks=1. 

The impact is quantified here:
https://cwiki.apache.org/confluence/display/KAFKA/An+analysis+of+the+impact+of+max.in.flight.requests.per.connection+and+acks+on+Producer+performance

However, we can't explain some of that data. Nor do we know the causes for some 
of the degradation. We would like to understand the performance of acks=all at 
the very minimum before making it the default producer setting.

Reporter: Apurva Mehta
Assignee: Apurva Mehta






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5795) Make the idempotent producer the default producer setting

[Apurva Mehta (JIRA)]

Apurva Mehta created KAFKA-5795:
---

 Summary: Make the idempotent producer the default producer setting
 Key: KAFKA-5795
 URL: https://issues.apache.org/jira/browse/KAFKA-5795
 Project: Kafka
  Issue Type: Improvement
Reporter: Apurva Mehta
Assignee: Apurva Mehta
 Fix For: 1.0.0


We would like to turn on idempotence by default. The KIP is here: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-185%3A+Make+exactly+once+in+order+delivery+per+partition+the+default+producer+setting



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5794) Introduce new idempotence mode to gracefully deal with topics on the older message format

[Apurva Mehta (JIRA)]

Apurva Mehta created KAFKA-5794:
---

 Summary: Introduce new idempotence mode to gracefully deal with 
topics on the older message format
 Key: KAFKA-5794
 URL: https://issues.apache.org/jira/browse/KAFKA-5794
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.11.0.0
Reporter: Apurva Mehta
Assignee: Apurva Mehta
 Fix For: 1.0.0


In the discussion of KIP-185: Make exactly once in order delivery per partition 
the default producer setting, it was realized that we don't have graceful 
handling when an idempotence-enabled producer is writing to a broker with a 
message format older than v2 (ie. the 0.11.0 message format). 

In particular, if we enable idempotence, any produce requests to topics with an 
older message format will fail with an UnsupportedVersionException. Thus if the 
idempotent producer was to be made the default, the out of the box producer 
would fail to produce when used with clusters which haven't upgraded the 
message format yet.

This is particularly problematic since the recommended upgrade path is to 
upgrade broker code while keeping the message format at the older version, then 
upgrade all clients, and only finally upgrade the message format on the server. 
With the current behavior, the middle step is actually untenable if we enable 
idempotence as the default.

More details available at: 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Exactly+Once+-+Dealing+with+older+message+formats+when+idempotence+is+enabled



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5793) Tighten up situations where OutOfOrderSequence may be returned

[Apurva Mehta (JIRA)]

Apurva Mehta created KAFKA-5793:
---

 Summary: Tighten up situations where OutOfOrderSequence may be 
returned
 Key: KAFKA-5793
 URL: https://issues.apache.org/jira/browse/KAFKA-5793
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.11.0.0
Reporter: Apurva Mehta
Assignee: Apurva Mehta
 Fix For: 1.0.0


Details of the problem are provided here: 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Exactly+Once+-+Solving+the+problem+of+spurious+OutOfOrderSequence+errors

A quick summary follows:

In the discussion of KIP-185: Make exactly once in order delivery per partition 
the default producer setting, the following point regarding the 
OutOfOrderSequenceException was raised:

1. The OutOfOrderSequenceException indicates that there has been data loss on 
the broker.. ie. a previously acknowledged message no longer exists. For most 
part, this should only occur in rare situations (simultaneous power outages, 
multiple disk losses, software bugs resulting in data corruption, etc.).
2. However, there is another perfectly normal scenario where data is removed: 
in particular, data could be deleted because it is old and crosses the 
retention threshold.
Hence, if a producer remains inactive for longer than a topic's retention 
period, we could get an OutOfOrderSequence which is a false positive: the data 
is removed through valid processes, and this isn't an error.
3. We would like to eliminate the possibility of getting spurious 
OutOfOrderSequenceExceptions – when you get it, it should always mean data loss 
and should be taken very seriously. 




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

[Becket Qin]

Hi Jason,

I see what you mean. That makes sense. So in the above case after the
producer resets PID, when it retry batch_0_tp1, the batch will still have
the old PID even if the producer has already got a new PID.

@Jun, do you mean max(remaining delivery.timeout.ms, request.timeout.ms)
instead of min(remaining delivery.timeout.ms, request.timeout.ms)?

Thanks,

Jiangjie (Becket) Qin

On Fri, Aug 25, 2017 at 9:34 AM, Jun Rao  wrote:

> Hi, Becket,
>
> Good point on expiring inflight requests. Perhaps we can expire an inflight
> request after min(remaining delivery.timeout.ms, request.timeout.ms). This
> way, if a user sets a high delivery.timeout.ms, we can still recover from
> broker power outage sooner.
>
> Thanks,
>
> Jun
>
> On Thu, Aug 24, 2017 at 12:52 PM, Becket Qin  wrote:
>
> > Hi Jason,
> >
> > delivery.timeout.ms sounds good to me.
> >
> > I was referring to the case that we are resetting the PID/sequence after
> > expire a batch. This is more about the sending the batches after the
> > expired batch.
> >
> > The scenario being discussed is expiring one of the batches in a
> in-flight
> > request and retry the other batches in the that in-flight request. So
> > consider the following case:
> > 1. Producer sends request_0 with two batches (batch_0_tp0 and
> batch_0_tp1).
> > 2. Broker receives the request enqueued the request to the log.
> > 3. Before the producer receives the response from the broker, batch_0_tp0
> > expires. The producer will expire batch_0_tp0 immediately, resets PID,
> and
> > then resend batch_0_tp1, and maybe send batch_1_tp0 (i.e. the next batch
> to
> > the expired batch) as well.
> >
> > For batch_0_tp1, it is OK to reuse PID and and sequence number. The
> problem
> > is for batch_1_tp0, If we reuse the same PID and the broker has already
> > appended batch_0_tp0, the broker will think batch_1_tp0 is a duplicate
> with
> > the same sequence number. As a result broker will drop batch_0_tp1. That
> is
> > why we have to either bump up sequence number or reset PID. To avoid this
> > complexity, I was suggesting not expire the in-flight batch immediately,
> > but wait for the produce response. If the batch has been successfully
> > appended, we do not expire it. Otherwise, we expire it.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> >
> > On Thu, Aug 24, 2017 at 11:26 AM, Jason Gustafson 
> > wrote:
> >
> > > @Becket
> > >
> > > Good point about unnecessarily resetting the PID in cases where we know
> > the
> > > request has failed. Might be worth opening a JIRA to try and improve
> > this.
> > >
> > > So if we expire the batch prematurely and resend all
> > > > the other batches in the same request, chances are there will be
> > > > duplicates. If we wait for the response instead, it is less likely to
> > > > introduce duplicates, and we may not need to reset the PID.
> > >
> > >
> > > Not sure I follow this. Are you assuming that we change the batch
> > > PID/sequence of the retried batches after resetting the PID? I think we
> > > probably need to ensure that when we retry a batch, we always use the
> > same
> > > PID/sequence.
> > >
> > > By the way, as far as naming, `max.message.delivery.wait.ms` is quite
> a
> > > mouthful. Could we shorten it? Perhaps `delivery.timeout.ms`?
> > >
> > > -Jason
> > >
> > > On Wed, Aug 23, 2017 at 8:51 PM, Becket Qin 
> > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > If TCP timeout is longer than request.timeout.ms, the producer will
> > > always
> > > > hit request.timeout.ms before hitting TCP timeout, right? That is
> why
> > we
> > > > added request.timeout.ms in the first place.
> > > >
> > > > You are right. Currently we are reset the PID and resend the batches
> to
> > > > avoid OutOfOrderSequenceException when the expired batches are in
> > retry.
> > > >
> > > > This does not distinguish the reasons that caused the retry. There
> are
> > > two
> > > > cases:
> > > > 1. If the batch was in retry because it received an error response
> > (e.g.
> > > > NotLeaderForPartition), we actually don't need to reset PID in this
> > case
> > > > because we know that broker did not accept it.
> > > > 2. If the batch was in retry because it hit a timeout earlier, then
> we
> > > > should reset the PID (or optimistically send and only reset PID when
> > > > receive OutOfOrderSequenceException?)
> > > > Case 1 is probably the most common case, so it looks that we are
> > > resetting
> > > > the PID more often than necessary. But because in case 1 the broker
> > does
> > > > not have the batch, there isn't much impact on resting PID and resend
> > > other
> > > > than the additional round trip.
> > > >
> > > > Now we are introducing another case:
> > > > 3. A batch is in retry because we expired an in-flight request before
> > it
> > > > hits request.timeout.ms.
> > > >
> > > > The difference between 2 and 3 is that in case 3 likely the broker
> has
> > 

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

[Don Bosco Durai]

Jason, thanks for the clarification.

Bosco


On 8/25/17, 4:59 PM, "Jason Gustafson"  wrote:

Hey Don,

That is not actually part of the KIP. It was a (somewhat pedantic) example
used to illustrate how the kafka principal semantics could be applied to
authorizers which understood group-level ACLs. The key point is this:
although a principal is identified only by its type and name, the
KafkaPrincipal can be used to represent relations to other principals. In
this case, we have a user principal which is related to a group principal
through the UserPrincipalAndGroup object. A GroupAuthorizer could then
leverage this relation. As you suggest, a true implementation would allow
multiple groups.

I will add a note to the KIP to emphasize that this is just an example.

Thanks,
Jason

On Fri, Aug 25, 2017 at 4:37 PM, Don Bosco Durai  wrote:

> Jason, thanks for confirming that. Since there are existing custom
> plugins, we might have to give enough time for them to start using the
> newer interface.
>
> I quickly glanced over the KIP, it looks good. Here is one comment:
>
> ---
> In the future, we may add support for groups to Kafka. This was brought up
> in the KIP-111 discussion. To support this, we can provide a groupId()
> method in KafkaPrincipal which defaults to a null value or an empty 
string.
> Extensions can override this just as before. Also note that it is still
> possible for the Authorizer implementation to derive its own group
> information for enforcement.
> class UserPrincipalAndGroup extends KafkaPrincipal {
>   private final String userId;
>   private final String groupId;
> ---
>
> We should assume that users might belong to multiple groups.
>
> Also, not sure what the below method is really doing?
> ---
> class UserPrincipalAndGroup extends KafkaPrincipal {
>
> public KafkaPrincipal group() {
> return new KafkaPrincipal(KafkaPrincipal.GROUP_TYPE, groupId);
>   }
> ---
> Thanks
>
> Bosco
>
>
>
> On 8/25/17, 4:11 PM, "Jason Gustafson"  wrote:
>
> Hi Don,
>
> I don't think so. We are not making any changes to the Authorizer
> interface
> itself. The KafkaPrincipal object does not change either, though we 
now
> explicitly allow it to be extended. That means you have to exercise a
> little caution when combining a custom PrincipalBuilder with a custom
> Authorizer. For the default principal builder shipped with Kafka, it
> will
> work the same as it currently does. Old implementations of
> PrincipalBuilder
> will also continue to work exactly as they do now, but please note
> that I
> am proposing to deprecate this interface. It will still be supported 
in
> 1.0.0, but we may remove it in a future major release.
>
> -Jason
>
> On Fri, Aug 25, 2017 at 3:51 PM, Don Bosco Durai 
> wrote:
>
> > Jason
> >
> > Do you anticipate any backward compatibility issues with existing
> custom
> > implementation of the authorization interface/plugins?
> >
> > Thanks
> >
> > Bosco
> >
> >
> > On 8/25/17, 3:22 PM, "Jason Gustafson"  wrote:
> >
> > No problem. I'll add a note to the KIP to emphasize that we will
> use
> > the
> > same object built by the KafkaPrincipalBuilder in the Session
> object
> > passed
> > to the authorizer.
> >
> > -Jason
> >
> > On Fri, Aug 25, 2017 at 3:17 PM, Mayuresh Gharat <
> > gharatmayures...@gmail.com
> > > wrote:
> >
> > > Perfect.
> > > As long as there is a way we can access the originally created
> > Principal in
> > > the Authorizer, it would solve the KIP-111 issue.
> > >
> > > This is really helpful, thanks again.
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Fri, Aug 25, 2017 at 3:13 PM, Jason Gustafson <
> ja...@confluent.io
> > >
> > > wrote:
> > >
> > > > Hi Mayuresh,
> > > >
> > > > To clarify, the intention is to use the KafkaPrincipal 
object
> > built by
> > > the
> > > > KafkaPrincipalBuilder inside the Session. So we would remove
> the
> > logic to
> > > > construct a new KafkaPrincipal using only the name from the
> > Principal.
> > > Then
> > > > it should be possible to pass the 

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

[Jason Gustafson]

Hey Don,

That is not actually part of the KIP. It was a (somewhat pedantic) example
used to illustrate how the kafka principal semantics could be applied to
authorizers which understood group-level ACLs. The key point is this:
although a principal is identified only by its type and name, the
KafkaPrincipal can be used to represent relations to other principals. In
this case, we have a user principal which is related to a group principal
through the UserPrincipalAndGroup object. A GroupAuthorizer could then
leverage this relation. As you suggest, a true implementation would allow
multiple groups.

I will add a note to the KIP to emphasize that this is just an example.

Thanks,
Jason

On Fri, Aug 25, 2017 at 4:37 PM, Don Bosco Durai  wrote:

> Jason, thanks for confirming that. Since there are existing custom
> plugins, we might have to give enough time for them to start using the
> newer interface.
>
> I quickly glanced over the KIP, it looks good. Here is one comment:
>
> ---
> In the future, we may add support for groups to Kafka. This was brought up
> in the KIP-111 discussion. To support this, we can provide a groupId()
> method in KafkaPrincipal which defaults to a null value or an empty string.
> Extensions can override this just as before. Also note that it is still
> possible for the Authorizer implementation to derive its own group
> information for enforcement.
> class UserPrincipalAndGroup extends KafkaPrincipal {
>   private final String userId;
>   private final String groupId;
> ---
>
> We should assume that users might belong to multiple groups.
>
> Also, not sure what the below method is really doing?
> ---
> class UserPrincipalAndGroup extends KafkaPrincipal {
>
> public KafkaPrincipal group() {
> return new KafkaPrincipal(KafkaPrincipal.GROUP_TYPE, groupId);
>   }
> ---
> Thanks
>
> Bosco
>
>
>
> On 8/25/17, 4:11 PM, "Jason Gustafson"  wrote:
>
> Hi Don,
>
> I don't think so. We are not making any changes to the Authorizer
> interface
> itself. The KafkaPrincipal object does not change either, though we now
> explicitly allow it to be extended. That means you have to exercise a
> little caution when combining a custom PrincipalBuilder with a custom
> Authorizer. For the default principal builder shipped with Kafka, it
> will
> work the same as it currently does. Old implementations of
> PrincipalBuilder
> will also continue to work exactly as they do now, but please note
> that I
> am proposing to deprecate this interface. It will still be supported in
> 1.0.0, but we may remove it in a future major release.
>
> -Jason
>
> On Fri, Aug 25, 2017 at 3:51 PM, Don Bosco Durai 
> wrote:
>
> > Jason
> >
> > Do you anticipate any backward compatibility issues with existing
> custom
> > implementation of the authorization interface/plugins?
> >
> > Thanks
> >
> > Bosco
> >
> >
> > On 8/25/17, 3:22 PM, "Jason Gustafson"  wrote:
> >
> > No problem. I'll add a note to the KIP to emphasize that we will
> use
> > the
> > same object built by the KafkaPrincipalBuilder in the Session
> object
> > passed
> > to the authorizer.
> >
> > -Jason
> >
> > On Fri, Aug 25, 2017 at 3:17 PM, Mayuresh Gharat <
> > gharatmayures...@gmail.com
> > > wrote:
> >
> > > Perfect.
> > > As long as there is a way we can access the originally created
> > Principal in
> > > the Authorizer, it would solve the KIP-111 issue.
> > >
> > > This is really helpful, thanks again.
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Fri, Aug 25, 2017 at 3:13 PM, Jason Gustafson <
> ja...@confluent.io
> > >
> > > wrote:
> > >
> > > > Hi Mayuresh,
> > > >
> > > > To clarify, the intention is to use the KafkaPrincipal object
> > built by
> > > the
> > > > KafkaPrincipalBuilder inside the Session. So we would remove
> the
> > logic to
> > > > construct a new KafkaPrincipal using only the name from the
> > Principal.
> > > Then
> > > > it should be possible to pass the `AuthzPrincipal` to the
> > underlying
> > > > library through the `Extended_Plugged_In_Class` as you've
> suggested
> > > above.
> > > > Is that reasonable for this use case?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > > On Fri, Aug 25, 2017 at 2:44 PM, Mayuresh Gharat <
> > > > gharatmayures...@gmail.com
> > > > > wrote:
> > > >
> > > > > Hi Jason,
> > > > >
> > > > > Thanks for the replies.
> > > > >
> > > > > I think it would be better to discuss with an example that
> we
> > 

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

[Don Bosco Durai]

Jason, thanks for confirming that. Since there are existing custom plugins, we 
might have to give enough time for them to start using the newer interface.

I quickly glanced over the KIP, it looks good. Here is one comment:

---
In the future, we may add support for groups to Kafka. This was brought up in 
the KIP-111 discussion. To support this, we can provide a groupId() method in 
KafkaPrincipal which defaults to a null value or an empty string. Extensions 
can override this just as before. Also note that it is still possible for the 
Authorizer implementation to derive its own group information for enforcement.
class UserPrincipalAndGroup extends KafkaPrincipal {
  private final String userId;
  private final String groupId;
---

We should assume that users might belong to multiple groups.

Also, not sure what the below method is really doing?
---
class UserPrincipalAndGroup extends KafkaPrincipal {

public KafkaPrincipal group() {
return new KafkaPrincipal(KafkaPrincipal.GROUP_TYPE, groupId);
  }
---
Thanks

Bosco



On 8/25/17, 4:11 PM, "Jason Gustafson"  wrote:

Hi Don,

I don't think so. We are not making any changes to the Authorizer interface
itself. The KafkaPrincipal object does not change either, though we now
explicitly allow it to be extended. That means you have to exercise a
little caution when combining a custom PrincipalBuilder with a custom
Authorizer. For the default principal builder shipped with Kafka, it will
work the same as it currently does. Old implementations of PrincipalBuilder
will also continue to work exactly as they do now, but please note that I
am proposing to deprecate this interface. It will still be supported in
1.0.0, but we may remove it in a future major release.

-Jason

On Fri, Aug 25, 2017 at 3:51 PM, Don Bosco Durai  wrote:

> Jason
>
> Do you anticipate any backward compatibility issues with existing custom
> implementation of the authorization interface/plugins?
>
> Thanks
>
> Bosco
>
>
> On 8/25/17, 3:22 PM, "Jason Gustafson"  wrote:
>
> No problem. I'll add a note to the KIP to emphasize that we will use
> the
> same object built by the KafkaPrincipalBuilder in the Session object
> passed
> to the authorizer.
>
> -Jason
>
> On Fri, Aug 25, 2017 at 3:17 PM, Mayuresh Gharat <
> gharatmayures...@gmail.com
> > wrote:
>
> > Perfect.
> > As long as there is a way we can access the originally created
> Principal in
> > the Authorizer, it would solve the KIP-111 issue.
> >
> > This is really helpful, thanks again.
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Fri, Aug 25, 2017 at 3:13 PM, Jason Gustafson  >
> > wrote:
> >
> > > Hi Mayuresh,
> > >
> > > To clarify, the intention is to use the KafkaPrincipal object
> built by
> > the
> > > KafkaPrincipalBuilder inside the Session. So we would remove the
> logic to
> > > construct a new KafkaPrincipal using only the name from the
> Principal.
> > Then
> > > it should be possible to pass the `AuthzPrincipal` to the
> underlying
> > > library through the `Extended_Plugged_In_Class` as you've 
suggested
> > above.
> > > Is that reasonable for this use case?
> > >
> > > Thanks,
> > > Jason
> > >
> > >
> > > On Fri, Aug 25, 2017 at 2:44 PM, Mayuresh Gharat <
> > > gharatmayures...@gmail.com
> > > > wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > Thanks for the replies.
> > > >
> > > > I think it would be better to discuss with an example that we
> were
> > trying
> > > > to address with KIP-111 and see if the current mentioned
> solution would
> > > > address it.
> > > >
> > > > Let's consider a third party library called authz_lib that is
> provided
> > by
> > > > some Security team at  some company.
> > > >
> > > >- When we call authz_lib.createPrincipal(X509_cert), it would
> > return
> > > an
> > > >AuthzPrincipal that implements Java.Security.Principal.
> > > >
> > > >
> > > >- The authz_lib also provides an checkAccess() call that
> takes
> > in
> > > 3
> > > >parameters :
> > > >   - authz_principal
> > > >   - operation type ("Read", "Write"...)
> > > >   - resource (for simplicity lets consider it as a 
TopicName)
> > > >
> > > >
> > > >- The AuthzPrincipal looks like this :
> 

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

[Don Bosco Durai]

Jason

Do you anticipate any backward compatibility issues with existing custom 
implementation of the authorization interface/plugins?

Thanks

Bosco


On 8/25/17, 3:22 PM, "Jason Gustafson"  wrote:

No problem. I'll add a note to the KIP to emphasize that we will use the
same object built by the KafkaPrincipalBuilder in the Session object passed
to the authorizer.

-Jason

On Fri, Aug 25, 2017 at 3:17 PM, Mayuresh Gharat  wrote:

> Perfect.
> As long as there is a way we can access the originally created Principal 
in
> the Authorizer, it would solve the KIP-111 issue.
>
> This is really helpful, thanks again.
>
> Thanks,
>
> Mayuresh
>
> On Fri, Aug 25, 2017 at 3:13 PM, Jason Gustafson 
> wrote:
>
> > Hi Mayuresh,
> >
> > To clarify, the intention is to use the KafkaPrincipal object built by
> the
> > KafkaPrincipalBuilder inside the Session. So we would remove the logic 
to
> > construct a new KafkaPrincipal using only the name from the Principal.
> Then
> > it should be possible to pass the `AuthzPrincipal` to the underlying
> > library through the `Extended_Plugged_In_Class` as you've suggested
> above.
> > Is that reasonable for this use case?
> >
> > Thanks,
> > Jason
> >
> >
> > On Fri, Aug 25, 2017 at 2:44 PM, Mayuresh Gharat <
> > gharatmayures...@gmail.com
> > > wrote:
> >
> > > Hi Jason,
> > >
> > > Thanks for the replies.
> > >
> > > I think it would be better to discuss with an example that we were
> trying
> > > to address with KIP-111 and see if the current mentioned solution 
would
> > > address it.
> > >
> > > Let's consider a third party library called authz_lib that is provided
> by
> > > some Security team at  some company.
> > >
> > >- When we call authz_lib.createPrincipal(X509_cert), it would
> return
> > an
> > >AuthzPrincipal that implements Java.Security.Principal.
> > >
> > >
> > >- The authz_lib also provides an checkAccess() call that takes
> in
> > 3
> > >parameters :
> > >   - authz_principal
> > >   - operation type ("Read", "Write"...)
> > >   - resource (for simplicity lets consider it as a TopicName)
> > >
> > >
> > >- The AuthzPrincipal looks like this :
> > >
> > > class AuthzPrincipal implements java.security.Principal
> > > {
> > > String name;
> > > String field1;
> > > Object field2;
> > > Object field3;
> > > .//Some third party logic..
> > > }
> > >
> > >
> > >- In PrincipalBuilder.buildPrincipal() would return AuthzPrincipal
> as
> > >follows :
> > >
> > > public Principal buildPrincipal(...)
> > > {
> > > ..
> > > X509Certificate x509Cert = session.getCert(..);
> > > return authz_lib.createPrincipal(x509Cert);
> > > }
> > >
> > >
> > >- The custom Authorizer (lets call it CustomAuthzAuthorizer), we
> would
> > >use the checkAccess() function provided by the authz_lib as follows
> :
> > >
> > > public class CustomAuthzAuthorizer implements Authorizer
> > > {
> > > .
> > > public boolean authorize(.)
> > > {
> > >AuthzPrincipal authz_principal = (AuthzPrincipal)
> > > session.getPrincipal();
> > > return authz_lib.checkAccess(authz_principal, "Read", "topicX");
> > > }
> > > ..
> > > }
> > >
> > >
> > >- The issue with current implementation is that in
> > >processCompletedReceives() in SocketServer we create a
> KafkaPrincipal
> > >that just extracts the name from AuthzPrincipal as follows :
> > >
> > > session = RequestChannel.Session(new
> > > KafkaPrincipal(KafkaPrincipal.USER_TYPE,
> > > *openOrClosingChannel.principal.getName*),
> > > openOrClosingChannel.socketAddress)
> > >
> > > So the "AuthzPrincipal authz_principal = (AuthzPrincipal)
> > > session.getPrincipal()" call in the CustomAuthzAuthorizer would error
> > > out because we are trying to cast a KafkaPrincipal to AuthzPrincipal.
> > >
> > >
> > >
> > > In your reply when you said that :
> > >
> > > The KIP says that a user can have a class that extends KafkaPrincipal.
> > > Would this extended class be used when constructing the Session object
> > > in the SocketServer instead of constructing a new KafkaPrincipal?
> > >
> > > Yes, that's correct. We want to allow the authorizer to be able to
> > leverage
> > > > additional information from the authentication layer.
> > >
> > >
> > > Would it make sense to make this extended class pluggable and when
> > > 

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

[Jason Gustafson]

No problem. I'll add a note to the KIP to emphasize that we will use the
same object built by the KafkaPrincipalBuilder in the Session object passed
to the authorizer.

-Jason

On Fri, Aug 25, 2017 at 3:17 PM, Mayuresh Gharat  wrote:

> Perfect.
> As long as there is a way we can access the originally created Principal in
> the Authorizer, it would solve the KIP-111 issue.
>
> This is really helpful, thanks again.
>
> Thanks,
>
> Mayuresh
>
> On Fri, Aug 25, 2017 at 3:13 PM, Jason Gustafson 
> wrote:
>
> > Hi Mayuresh,
> >
> > To clarify, the intention is to use the KafkaPrincipal object built by
> the
> > KafkaPrincipalBuilder inside the Session. So we would remove the logic to
> > construct a new KafkaPrincipal using only the name from the Principal.
> Then
> > it should be possible to pass the `AuthzPrincipal` to the underlying
> > library through the `Extended_Plugged_In_Class` as you've suggested
> above.
> > Is that reasonable for this use case?
> >
> > Thanks,
> > Jason
> >
> >
> > On Fri, Aug 25, 2017 at 2:44 PM, Mayuresh Gharat <
> > gharatmayures...@gmail.com
> > > wrote:
> >
> > > Hi Jason,
> > >
> > > Thanks for the replies.
> > >
> > > I think it would be better to discuss with an example that we were
> trying
> > > to address with KIP-111 and see if the current mentioned solution would
> > > address it.
> > >
> > > Let's consider a third party library called authz_lib that is provided
> by
> > > some Security team at  some company.
> > >
> > >- When we call authz_lib.createPrincipal(X509_cert), it would
> return
> > an
> > >AuthzPrincipal that implements Java.Security.Principal.
> > >
> > >
> > >- The authz_lib also provides an checkAccess() call that takes
> in
> > 3
> > >parameters :
> > >   - authz_principal
> > >   - operation type ("Read", "Write"...)
> > >   - resource (for simplicity lets consider it as a TopicName)
> > >
> > >
> > >- The AuthzPrincipal looks like this :
> > >
> > > class AuthzPrincipal implements java.security.Principal
> > > {
> > > String name;
> > > String field1;
> > > Object field2;
> > > Object field3;
> > > .//Some third party logic..
> > > }
> > >
> > >
> > >- In PrincipalBuilder.buildPrincipal() would return AuthzPrincipal
> as
> > >follows :
> > >
> > > public Principal buildPrincipal(...)
> > > {
> > > ..
> > > X509Certificate x509Cert = session.getCert(..);
> > > return authz_lib.createPrincipal(x509Cert);
> > > }
> > >
> > >
> > >- The custom Authorizer (lets call it CustomAuthzAuthorizer), we
> would
> > >use the checkAccess() function provided by the authz_lib as follows
> :
> > >
> > > public class CustomAuthzAuthorizer implements Authorizer
> > > {
> > > .
> > > public boolean authorize(.)
> > > {
> > >AuthzPrincipal authz_principal = (AuthzPrincipal)
> > > session.getPrincipal();
> > > return authz_lib.checkAccess(authz_principal, "Read", "topicX");
> > > }
> > > ..
> > > }
> > >
> > >
> > >- The issue with current implementation is that in
> > >processCompletedReceives() in SocketServer we create a
> KafkaPrincipal
> > >that just extracts the name from AuthzPrincipal as follows :
> > >
> > > session = RequestChannel.Session(new
> > > KafkaPrincipal(KafkaPrincipal.USER_TYPE,
> > > *openOrClosingChannel.principal.getName*),
> > > openOrClosingChannel.socketAddress)
> > >
> > > So the "AuthzPrincipal authz_principal = (AuthzPrincipal)
> > > session.getPrincipal()" call in the CustomAuthzAuthorizer would error
> > > out because we are trying to cast a KafkaPrincipal to AuthzPrincipal.
> > >
> > >
> > >
> > > In your reply when you said that :
> > >
> > > The KIP says that a user can have a class that extends KafkaPrincipal.
> > > Would this extended class be used when constructing the Session object
> > > in the SocketServer instead of constructing a new KafkaPrincipal?
> > >
> > > Yes, that's correct. We want to allow the authorizer to be able to
> > leverage
> > > > additional information from the authentication layer.
> > >
> > >
> > > Would it make sense to make this extended class pluggable and when
> > > constructing the Session object in SocketServer check if a plugin is
> > > defined and use it and if not use the default KafkaPrincipal something
> > like
> > > :
> > >
> > > if (getConfig("principal.pluggedIn.class").isDefined())
> > > //"principal.pluggedIn.class"
> > > is just an example name for the config
> > > {
> > > session = RequestChannel.Session(*Extended_Plugged_In_Class*,
> > > openOrClosingChannel.socketAddress)
> > > }
> > > else
> > > {
> > > session = RequestChannel.Session(new KafkaPrincipal(KafkaPrincipal.
> > > USER_TYPE,
> > > *openOrClosingChannel.principal.getName*),
> > > openOrClosingChannel.socketAddress)
> > > }
> > >
> > > This would solve the issue above as follows :
> > >
> > > We can have something like :
> > > public class Extended_Plugged_In_Class extends 

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

[Mayuresh Gharat]

Perfect.
As long as there is a way we can access the originally created Principal in
the Authorizer, it would solve the KIP-111 issue.

This is really helpful, thanks again.

Thanks,

Mayuresh

On Fri, Aug 25, 2017 at 3:13 PM, Jason Gustafson  wrote:

> Hi Mayuresh,
>
> To clarify, the intention is to use the KafkaPrincipal object built by the
> KafkaPrincipalBuilder inside the Session. So we would remove the logic to
> construct a new KafkaPrincipal using only the name from the Principal. Then
> it should be possible to pass the `AuthzPrincipal` to the underlying
> library through the `Extended_Plugged_In_Class` as you've suggested above.
> Is that reasonable for this use case?
>
> Thanks,
> Jason
>
>
> On Fri, Aug 25, 2017 at 2:44 PM, Mayuresh Gharat <
> gharatmayures...@gmail.com
> > wrote:
>
> > Hi Jason,
> >
> > Thanks for the replies.
> >
> > I think it would be better to discuss with an example that we were trying
> > to address with KIP-111 and see if the current mentioned solution would
> > address it.
> >
> > Let's consider a third party library called authz_lib that is provided by
> > some Security team at  some company.
> >
> >- When we call authz_lib.createPrincipal(X509_cert), it would return
> an
> >AuthzPrincipal that implements Java.Security.Principal.
> >
> >
> >- The authz_lib also provides an checkAccess() call that takes in
> 3
> >parameters :
> >   - authz_principal
> >   - operation type ("Read", "Write"...)
> >   - resource (for simplicity lets consider it as a TopicName)
> >
> >
> >- The AuthzPrincipal looks like this :
> >
> > class AuthzPrincipal implements java.security.Principal
> > {
> > String name;
> > String field1;
> > Object field2;
> > Object field3;
> > .//Some third party logic..
> > }
> >
> >
> >- In PrincipalBuilder.buildPrincipal() would return AuthzPrincipal as
> >follows :
> >
> > public Principal buildPrincipal(...)
> > {
> > ..
> > X509Certificate x509Cert = session.getCert(..);
> > return authz_lib.createPrincipal(x509Cert);
> > }
> >
> >
> >- The custom Authorizer (lets call it CustomAuthzAuthorizer), we would
> >use the checkAccess() function provided by the authz_lib as follows :
> >
> > public class CustomAuthzAuthorizer implements Authorizer
> > {
> > .
> > public boolean authorize(.)
> > {
> >AuthzPrincipal authz_principal = (AuthzPrincipal)
> > session.getPrincipal();
> > return authz_lib.checkAccess(authz_principal, "Read", "topicX");
> > }
> > ..
> > }
> >
> >
> >- The issue with current implementation is that in
> >processCompletedReceives() in SocketServer we create a KafkaPrincipal
> >that just extracts the name from AuthzPrincipal as follows :
> >
> > session = RequestChannel.Session(new
> > KafkaPrincipal(KafkaPrincipal.USER_TYPE,
> > *openOrClosingChannel.principal.getName*),
> > openOrClosingChannel.socketAddress)
> >
> > So the "AuthzPrincipal authz_principal = (AuthzPrincipal)
> > session.getPrincipal()" call in the CustomAuthzAuthorizer would error
> > out because we are trying to cast a KafkaPrincipal to AuthzPrincipal.
> >
> >
> >
> > In your reply when you said that :
> >
> > The KIP says that a user can have a class that extends KafkaPrincipal.
> > Would this extended class be used when constructing the Session object
> > in the SocketServer instead of constructing a new KafkaPrincipal?
> >
> > Yes, that's correct. We want to allow the authorizer to be able to
> leverage
> > > additional information from the authentication layer.
> >
> >
> > Would it make sense to make this extended class pluggable and when
> > constructing the Session object in SocketServer check if a plugin is
> > defined and use it and if not use the default KafkaPrincipal something
> like
> > :
> >
> > if (getConfig("principal.pluggedIn.class").isDefined())
> > //"principal.pluggedIn.class"
> > is just an example name for the config
> > {
> > session = RequestChannel.Session(*Extended_Plugged_In_Class*,
> > openOrClosingChannel.socketAddress)
> > }
> > else
> > {
> > session = RequestChannel.Session(new KafkaPrincipal(KafkaPrincipal.
> > USER_TYPE,
> > *openOrClosingChannel.principal.getName*),
> > openOrClosingChannel.socketAddress)
> > }
> >
> > This would solve the issue above as follows :
> >
> > We can have something like :
> > public class Extended_Plugged_In_Class extends KafkaPrincipal
> > {
> > AuthzPrincipal authzPrincipal;
> >
> > public Extended_Plugged_In_Class(., AuthzPrincipal principal)
> > {
> > super(...);
> > authzPrincipal = principal;
> >
> > }
> >
> > ..
> >
> > public AuthzPrincipal getAuthzPrincipal()
> > {
> > return authzPrincipal;
> > }
> > }
> >
> > In the CustomAuthzAuthorizer we could do something like :
> >
> > public class CustomAuthzAuthorizer implements Authorizer
> > {
> > .
> > public boolean authorize(.)
> > {
> > Extended_Plugged_In_Class  extended_Kafka_Principal =
> > 

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

[Jason Gustafson]

Hi Mayuresh,

To clarify, the intention is to use the KafkaPrincipal object built by the
KafkaPrincipalBuilder inside the Session. So we would remove the logic to
construct a new KafkaPrincipal using only the name from the Principal. Then
it should be possible to pass the `AuthzPrincipal` to the underlying
library through the `Extended_Plugged_In_Class` as you've suggested above.
Is that reasonable for this use case?

Thanks,
Jason


On Fri, Aug 25, 2017 at 2:44 PM, Mayuresh Gharat  wrote:

> Hi Jason,
>
> Thanks for the replies.
>
> I think it would be better to discuss with an example that we were trying
> to address with KIP-111 and see if the current mentioned solution would
> address it.
>
> Let's consider a third party library called authz_lib that is provided by
> some Security team at  some company.
>
>- When we call authz_lib.createPrincipal(X509_cert), it would return an
>AuthzPrincipal that implements Java.Security.Principal.
>
>
>- The authz_lib also provides an checkAccess() call that takes in 3
>parameters :
>   - authz_principal
>   - operation type ("Read", "Write"...)
>   - resource (for simplicity lets consider it as a TopicName)
>
>
>- The AuthzPrincipal looks like this :
>
> class AuthzPrincipal implements java.security.Principal
> {
> String name;
> String field1;
> Object field2;
> Object field3;
> .//Some third party logic..
> }
>
>
>- In PrincipalBuilder.buildPrincipal() would return AuthzPrincipal as
>follows :
>
> public Principal buildPrincipal(...)
> {
> ..
> X509Certificate x509Cert = session.getCert(..);
> return authz_lib.createPrincipal(x509Cert);
> }
>
>
>- The custom Authorizer (lets call it CustomAuthzAuthorizer), we would
>use the checkAccess() function provided by the authz_lib as follows :
>
> public class CustomAuthzAuthorizer implements Authorizer
> {
> .
> public boolean authorize(.)
> {
>AuthzPrincipal authz_principal = (AuthzPrincipal)
> session.getPrincipal();
> return authz_lib.checkAccess(authz_principal, "Read", "topicX");
> }
> ..
> }
>
>
>- The issue with current implementation is that in
>processCompletedReceives() in SocketServer we create a KafkaPrincipal
>that just extracts the name from AuthzPrincipal as follows :
>
> session = RequestChannel.Session(new
> KafkaPrincipal(KafkaPrincipal.USER_TYPE,
> *openOrClosingChannel.principal.getName*),
> openOrClosingChannel.socketAddress)
>
> So the "AuthzPrincipal authz_principal = (AuthzPrincipal)
> session.getPrincipal()" call in the CustomAuthzAuthorizer would error
> out because we are trying to cast a KafkaPrincipal to AuthzPrincipal.
>
>
>
> In your reply when you said that :
>
> The KIP says that a user can have a class that extends KafkaPrincipal.
> Would this extended class be used when constructing the Session object
> in the SocketServer instead of constructing a new KafkaPrincipal?
>
> Yes, that's correct. We want to allow the authorizer to be able to leverage
> > additional information from the authentication layer.
>
>
> Would it make sense to make this extended class pluggable and when
> constructing the Session object in SocketServer check if a plugin is
> defined and use it and if not use the default KafkaPrincipal something like
> :
>
> if (getConfig("principal.pluggedIn.class").isDefined())
> //"principal.pluggedIn.class"
> is just an example name for the config
> {
> session = RequestChannel.Session(*Extended_Plugged_In_Class*,
> openOrClosingChannel.socketAddress)
> }
> else
> {
> session = RequestChannel.Session(new KafkaPrincipal(KafkaPrincipal.
> USER_TYPE,
> *openOrClosingChannel.principal.getName*),
> openOrClosingChannel.socketAddress)
> }
>
> This would solve the issue above as follows :
>
> We can have something like :
> public class Extended_Plugged_In_Class extends KafkaPrincipal
> {
> AuthzPrincipal authzPrincipal;
>
> public Extended_Plugged_In_Class(., AuthzPrincipal principal)
> {
> super(...);
> authzPrincipal = principal;
>
> }
>
> ..
>
> public AuthzPrincipal getAuthzPrincipal()
> {
> return authzPrincipal;
> }
> }
>
> In the CustomAuthzAuthorizer we could do something like :
>
> public class CustomAuthzAuthorizer implements Authorizer
> {
> .
> public boolean authorize(.)
> {
> Extended_Plugged_In_Class  extended_Kafka_Principal =
> (Extended_Plugged_In_Class)
> session.getPrincipal();
>AuthzPrincipal authz_principal =
> extended_Kafka_Principal.getAuthzPrincipal();
> return authz_lib.checkAccess(authz_principal, "Read", "topicX");
> }
> ..
> }
>
>
> Thanks,
>
> Mayuresh
>
> On Fri, Aug 25, 2017 at 11:53 AM, Jason Gustafson 
> wrote:
>
> > Hey Mayuresh,
> >
> > Thanks for the comments.
> >
> >- The KIP says that a user can have a class that extends
> KafkaPrincipal.
> > >Would this extended class be used when constructing the Session
> object
> > > in
> > >the 

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

[Mayuresh Gharat]

Hi Jason,

Thanks for the replies.

I think it would be better to discuss with an example that we were trying
to address with KIP-111 and see if the current mentioned solution would
address it.

Let's consider a third party library called authz_lib that is provided by
some Security team at  some company.

   - When we call authz_lib.createPrincipal(X509_cert), it would return an
   AuthzPrincipal that implements Java.Security.Principal.


   - The authz_lib also provides an checkAccess() call that takes in 3
   parameters :
  - authz_principal
  - operation type ("Read", "Write"...)
  - resource (for simplicity lets consider it as a TopicName)


   - The AuthzPrincipal looks like this :

class AuthzPrincipal implements java.security.Principal
{
String name;
String field1;
Object field2;
Object field3;
.//Some third party logic..
}


   - In PrincipalBuilder.buildPrincipal() would return AuthzPrincipal as
   follows :

public Principal buildPrincipal(...)
{
..
X509Certificate x509Cert = session.getCert(..);
return authz_lib.createPrincipal(x509Cert);
}


   - The custom Authorizer (lets call it CustomAuthzAuthorizer), we would
   use the checkAccess() function provided by the authz_lib as follows :

public class CustomAuthzAuthorizer implements Authorizer
{
.
public boolean authorize(.)
{
   AuthzPrincipal authz_principal = (AuthzPrincipal)
session.getPrincipal();
return authz_lib.checkAccess(authz_principal, "Read", "topicX");
}
..
}


   - The issue with current implementation is that in
   processCompletedReceives() in SocketServer we create a KafkaPrincipal
   that just extracts the name from AuthzPrincipal as follows :

session = RequestChannel.Session(new
KafkaPrincipal(KafkaPrincipal.USER_TYPE,
*openOrClosingChannel.principal.getName*),
openOrClosingChannel.socketAddress)

So the "AuthzPrincipal authz_principal = (AuthzPrincipal)
session.getPrincipal()" call in the CustomAuthzAuthorizer would error
out because we are trying to cast a KafkaPrincipal to AuthzPrincipal.



In your reply when you said that :

The KIP says that a user can have a class that extends KafkaPrincipal.
Would this extended class be used when constructing the Session object
in the SocketServer instead of constructing a new KafkaPrincipal?

Yes, that's correct. We want to allow the authorizer to be able to leverage
> additional information from the authentication layer.


Would it make sense to make this extended class pluggable and when
constructing the Session object in SocketServer check if a plugin is
defined and use it and if not use the default KafkaPrincipal something like
:

if (getConfig("principal.pluggedIn.class").isDefined())
//"principal.pluggedIn.class"
is just an example name for the config
{
session = RequestChannel.Session(*Extended_Plugged_In_Class*,
openOrClosingChannel.socketAddress)
}
else
{
session = RequestChannel.Session(new KafkaPrincipal(KafkaPrincipal.USER_TYPE,
*openOrClosingChannel.principal.getName*),
openOrClosingChannel.socketAddress)
}

This would solve the issue above as follows :

We can have something like :
public class Extended_Plugged_In_Class extends KafkaPrincipal
{
AuthzPrincipal authzPrincipal;

public Extended_Plugged_In_Class(., AuthzPrincipal principal)
{
super(...);
authzPrincipal = principal;

}

..

public AuthzPrincipal getAuthzPrincipal()
{
return authzPrincipal;
}
}

In the CustomAuthzAuthorizer we could do something like :

public class CustomAuthzAuthorizer implements Authorizer
{
.
public boolean authorize(.)
{
Extended_Plugged_In_Class  extended_Kafka_Principal =
(Extended_Plugged_In_Class)
session.getPrincipal();
   AuthzPrincipal authz_principal =
extended_Kafka_Principal.getAuthzPrincipal();
return authz_lib.checkAccess(authz_principal, "Read", "topicX");
}
..
}


Thanks,

Mayuresh

On Fri, Aug 25, 2017 at 11:53 AM, Jason Gustafson 
wrote:

> Hey Mayuresh,
>
> Thanks for the comments.
>
>- The KIP says that a user can have a class that extends KafkaPrincipal.
> >Would this extended class be used when constructing the Session object
> > in
> >the SocketServer instead of constructing a new KafkaPrincipal?
>
>
> Yes, that's correct. We want to allow the authorizer to be able to leverage
> additional information from the authentication layer.
>
> - The KIP says "A principal is always identifiable by a principal type
> >and a name. Nothing else should ever be required." This might not be
> > true
> >always, right? For example, we might have a custom third party ACL
> > library
> >that creates a custom Principal from the passed in cert (this is done
> in
> >PrincipalBuilder/KafkaPrincipalBuilder) and the custom Authorizer
> might
> >use this third party library to authorize using this custom Principal
> >object. The developer who is implementing the Kafka Authorizer should
> >not be caring about what the custom Principal 

Build failed in Jenkins: kafka-trunk-jdk8 #1946

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] KAFKA-5776; Add the Trogdor fault injection daemon

--
[...truncated 2.02 MB...]

org.apache.kafka.common.security.scram.ScramMessagesTest > 
invalidServerFirstMessage STARTED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
invalidServerFirstMessage PASSED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
validServerFinalMessage STARTED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
validServerFinalMessage PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryWithoutPasswordConfiguration STARTED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryWithoutPasswordConfiguration PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > testClientMode STARTED

org.apache.kafka.common.security.ssl.SslFactoryTest > testClientMode PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryConfiguration STARTED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryConfiguration PASSED

org.apache.kafka.common.security.kerberos.KerberosNameTest > testParse STARTED

org.apache.kafka.common.security.kerberos.KerberosNameTest > testParse PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testPrincipalNameCanContainSeparator STARTED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testPrincipalNameCanContainSeparator PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testEqualsAndHashCode STARTED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testEqualsAndHashCode PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameOverride STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameOverride PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingOptionValue 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingOptionValue PASSED

org.apache.kafka.common.security.JaasContextTest > testSingleOption STARTED

org.apache.kafka.common.security.JaasContextTest > testSingleOption PASSED

org.apache.kafka.common.security.JaasContextTest > 
testNumericOptionWithoutQuotes STARTED

org.apache.kafka.common.security.JaasContextTest > 
testNumericOptionWithoutQuotes PASSED

org.apache.kafka.common.security.JaasContextTest > testConfigNoOptions STARTED

org.apache.kafka.common.security.JaasContextTest > testConfigNoOptions PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithWrongListenerName STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithWrongListenerName PASSED

org.apache.kafka.common.security.JaasContextTest > testNumericOptionWithQuotes 
STARTED

org.apache.kafka.common.security.JaasContextTest > testNumericOptionWithQuotes 
PASSED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionValue STARTED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionValue PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingLoginModule 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingLoginModule PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingSemicolon STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingSemicolon PASSED

org.apache.kafka.common.security.JaasContextTest > testMultipleOptions STARTED

org.apache.kafka.common.security.JaasContextTest > testMultipleOptions PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForClientWithListenerName STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForClientWithListenerName PASSED

org.apache.kafka.common.security.JaasContextTest > testMultipleLoginModules 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMultipleLoginModules 
PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingControlFlag 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingControlFlag PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameAndFallback STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameAndFallback PASSED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionName STARTED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionName PASSED

org.apache.kafka.common.security.JaasContextTest > testControlFlag STARTED

org.apache.kafka.common.security.JaasContextTest > testControlFlag PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingUsernameSaslPlain STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingUsernameSaslPlain PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testValidSaslScramMechanisms STARTED

[jira] [Created] (KAFKA-5792) Transient failure in KafkaAdminClientTest.testHandleTimeout

[Apurva Mehta (JIRA)]

Apurva Mehta created KAFKA-5792:
---

 Summary: Transient failure in 
KafkaAdminClientTest.testHandleTimeout
 Key: KAFKA-5792
 URL: https://issues.apache.org/jira/browse/KAFKA-5792
 Project: Kafka
  Issue Type: Bug
Reporter: Apurva Mehta


The {{KafkaAdminClientTest.testHandleTimeout}} test occasionally fails with the 
following:

{noformat}
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node 
assignment.
at 
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at 
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at 
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:213)
at 
org.apache.kafka.clients.admin.KafkaAdminClientTest.testHandleTimeout(KafkaAdminClientTest.java:356)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting 
for a node assignment.
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : kafka-trunk-jdk8 #1945

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-5452) Aggressive log compaction ratio appears to have no negative effect on log-compacted topics

[Jeff Chao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Chao resolved KAFKA-5452.
--
Resolution: Resolved

Following up after a long while. After talking offline with [~wushujames], the 
original thought was to choose a sensible default in relation to disk I/O. I 
think it's best to leave this default and prevent assumptions on the underlying 
infrastructure. That way, operators are free to tune to their expectations. 
Closing this.

> Aggressive log compaction ratio appears to have no negative effect on 
> log-compacted topics
> --
>
> Key: KAFKA-5452
> URL: https://issues.apache.org/jira/browse/KAFKA-5452
> Project: Kafka
>  Issue Type: Improvement
>  Components: config, core, log
>Affects Versions: 0.10.2.0, 0.10.2.1
> Environment: Ubuntu Trusty (14.04.5), Oracle JDK 8
>Reporter: Jeff Chao
>  Labels: performance
> Attachments: 200mbs-dirty0-dirty-1-dirty05.png, 
> flame-graph-200mbs-dirty0.png, flame-graph-200mbs-dirty0.svg
>
>
> Some of our users are seeing unintuitive/unexpected behavior with 
> log-compacted topics where they receive multiple records for the same key 
> when consuming. This is a result of low throughput on log-compacted topics 
> such that conditions ({{min.cleanable.dirty.ratio = 0.5}}, default) aren't 
> met for compaction to kick in.
> This prompted us to test and tune {{min.cleanable.dirty.ratio}} in our 
> clusters. It appears that having more aggressive log compaction ratios don't 
> have negative effects on CPU and memory utilization. If this is truly the 
> case, we should consider changing the default from {{0.5}} to something more 
> aggressive.
> Setup:
> # 8 brokers
> # 5 zk nodes
> # 32 partitions on a topic
> # replication factor 3
> # log roll 3 hours
> # log segment bytes 1 GB
> # log retention 24 hours
> # all messages to a single key
> # all messages to a unique key
> # all messages to a bounded key range [0, 999]
> # {{min.cleanable.dirty.ratio}} per topic = {{0}}, {{0.5}}, and {{1}}
> # 200 MB/s sustained, produce and consume traffic
> Observations:
> We were able to verify log cleaner threads were performing work by checking 
> the logs and verifying the {{cleaner-offset-checkpoint}} file for all topics. 
> We also observed the log cleaner's {{time-since-last-run-ms}} metric was 
> normal, never going above the default of 15 seconds.
> Under-replicated partitions stayed steady, same for replication lag.
> Here's an example test run where we try out {{min.cleanable.dirty.ratio = 
> 0}}, {{min.cleanable.dirty.ratio = 1}}, and {{min.cleanable.dirty.ratio = 
> 0.5}}. Troughs in between the peaks represent zero traffic and reconfiguring 
> of topics.
> (200mbs-dirty-0-dirty1-dirty05.png attached)
> !200mbs-dirty0-dirty-1-dirty05.png|thumbnail!
> Memory utilization is fine, but more interestingly, CPU doesn't appear to 
> have much difference.
> To get more detail, here is a flame graph (raw svg attached) of the run for 
> {{min.cleanable.dirty.ratio = 0}}. The conservative and default ratio flame 
> graphs are equivalent.
> (flame-graph-200mbs-dirty0.png attached)
> !flame-graph-200mbs-dirty0.png|thumbnail!
> Notice that the majority of CPU is coming from:
> # SSL operations (on reads/writes)
> # KafkaApis::handleFetchRequest (ReplicaManager::fetchMessages)
> # KafkaApis::handleOffsetFetchRequest
> We also have examples from small scale test runs which show similar behavior 
> but with scaled down CPU usage.
> It seems counterintuitive that there's no apparent difference in CPU whether 
> it be aggressive or conservative compaction ratios, so we'd like to get some 
> thoughts from the community.
> We're looking for feedback on whether or not anyone else has experienced this 
> behavior before as well or, if CPU isn't affected, has anyone seen something 
> related instead.
> If this is true, then we'd be happy to discuss further and provide a patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3744: Add group-id to the metrics tags

[EtaCassiopeia]

GitHub user EtaCassiopeia opened a pull request:

https://github.com/apache/kafka/pull/3744

Add group-id to the metrics tags

It is better to have group-id in the JMX metrics. It would improve 
debuggability of systems:

```kafka_consumer_consumer_fetch_manager_metrics_test_0_records_lag{client_id="consumer-1",group_id="group-1",instance="service:8080",job="prometheus"}```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/EtaCassiopeia/kafka 0.11.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3744.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3744


commit 86dbbf73300e865b2a8f50f0f79272c3b64a94b9
Author: Mohsen Zainalpour 
Date:   2017-08-25T20:10:40Z

Add group-id to the metrics tags




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3397: KAFKA-5413: Port fix to 0.10.2 branch

[kelvinrutt]

Github user kelvinrutt closed the pull request at:

https://github.com/apache/kafka/pull/3397


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3728: Add group-id to the metrics tags

[EtaCassiopeia]

Github user EtaCassiopeia closed the pull request at:

https://github.com/apache/kafka/pull/3728


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3743: KAFKA-5494: enable idempotence with max.in.flight....

[apurvam]

GitHub user apurvam opened a pull request:

https://github.com/apache/kafka/pull/3743

KAFKA-5494: enable idempotence with max.in.flight.requests.per.connection > 
1

Here we introduce client and broker changes to support multiple inflight 
requests while still guaranteeing idempotence. Two major problems to be solved:

1. Sequence number management on the client when there are request 
failures. When a batch fails,  future inflight batches will also fail with 
`OutOfOrderSequenceException`. This must be handled on the client with 
intelligent sequence reassignment. We must also deal with the fatal failure of 
some batch: the future batches must get different sequence numbers when the 
come back.
2. On the broker, when we have multiple inflights, we can get duplicates of 
multiple old batches. With this patch, we retain the record metadata for 5 
older batches. 

I have added `TODO(reviewers)` comments for specific decisions in the code 
which are worth discussing.

TODO: 
1. Add more unit tests, especially for loading different snapshot versions 
correctly, more client side unit tests, more broker side tests to validate that 
we are caching the correct number of batches (some of this is already there).
2. Update the system tests to check for ordering. 
3. Run a tight loop of system tests. 
4. Add comments about the assumptions made around the network client 
semantics of send/receive.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apurvam/kafka 
KAFKA-5494-increase-max-in-flight-for-idempotent-producer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3743.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3743


commit 005eee527ab425d8e3d8678aad4b5305cde6ca08
Author: Apurva Mehta 
Date:   2017-08-12T00:25:06Z

Initial commit of client side changes with some tests

commit 63bf074a38ec3efef728863081805a36d9111038
Author: Apurva Mehta 
Date:   2017-08-17T23:49:10Z

Implemented broker side changes to cache extra metadata.

Todo:
  1) Write more unit tests.
  2) Handle deletion / retention / cleaning correctly.

commit 1ad49f30f03ff665f5657680cbcc5e045210ce45
Author: Apurva Mehta 
Date:   2017-08-23T00:10:39Z

Change the client side code so that the sequence numbers are assigned
and incremented during drain. If a batch is retried, it's sequence
number is unset during the completion handler. If the first inflight
batch returns an error, the next sequence to assign is reset to the last
ack'd sequence + 1.

commit d9b86b7cb8e7001a7d5fc42a2ec061ebd0332a6a
Author: Apurva Mehta 
Date:   2017-08-24T01:33:54Z

WIP

commit 9ff885fe6db7172d28ea8fe406972a7763c0a49d
Author: Apurva Mehta 
Date:   2017-08-25T06:23:50Z

Implemented log cleaning functionality with tests

commit 5508a194c74a8946a8451c01814324e6ba788cfe
Author: Apurva Mehta 
Date:   2017-08-25T19:27:03Z

Fix merge issues aftre rebasing onto trunk




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3699: Add the Trogdor fault injection daemon

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3699


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-5776) Add the Trogdor fault injection daemon

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-5776.
---
   Resolution: Fixed
Fix Version/s: 1.0.0

Issue resolved by pull request 3699
[https://github.com/apache/kafka/pull/3699]

> Add the Trogdor fault injection daemon
> --
>
> Key: KAFKA-5776
> URL: https://issues.apache.org/jira/browse/KAFKA-5776
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Colin P. McCabe
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

[Jason Gustafson]

Hey Mayuresh,

Thanks for the comments.

   - The KIP says that a user can have a class that extends KafkaPrincipal.
>Would this extended class be used when constructing the Session object
> in
>the SocketServer instead of constructing a new KafkaPrincipal?


Yes, that's correct. We want to allow the authorizer to be able to leverage
additional information from the authentication layer.

- The KIP says "A principal is always identifiable by a principal type
>and a name. Nothing else should ever be required." This might not be
> true
>always, right? For example, we might have a custom third party ACL
> library
>that creates a custom Principal from the passed in cert (this is done in
>PrincipalBuilder/KafkaPrincipalBuilder) and the custom Authorizer might
>use this third party library to authorize using this custom Principal
>object. The developer who is implementing the Kafka Authorizer should
>not be caring about what the custom Principal would look like and its
>details, since it will just pass it to the third party library in Kafka
>Authorizer's authorize() call.


I'm not sure I understand this. Are you saying that the authorizer and
principal builder are implemented by separate individuals? If the
authorizer doesn't understand how to identify the principal, then it
wouldn't work, right? Maybe I'm missing something?

Let me explain how I see this. The simple ACL authorizer that Kafka ships
with understands user principals as consisting of a type and a name. Any
principal builder that follows this assumption will work with the
SimpleAclAuthorizer. In some cases, the principal builder may provide
additional metadata in a KafkaPrincipal extension such as user groups or
roles. This information is not needed to identify the user principal, so
the builder is still compatible with the SimpleAclAuthorizer. It would also
be compatible with a RoleBasedAuthorizer which understood how to use the
role metadata provided by the KafkaPrincipal extension. Basically what we
would have is a user principal which is related to one or more role
principals through the KafkaPrincipal extension. Both user and role
principals are identifiable with a type and a name, so the ACL command tool
can then be used (perhaps with a custom authorizer) to define permissions
in either case.

On the other hand, if a user principal is identified by more than just its
name, then it is not compatible with the SimpleAclAuthorizer. This doesn't
necessarily rule out this use case. As long as the authorizer and the
principal builder both agree on how user principals are identified, then
they can still be used together. But I am explicitly leaving out support in
the ACL command tool for this use case in this KIP. This is mostly about
clarifying what is compatible with the authorization system that Kafka
ships with. Of course we can always reconsider it in the future.

Thanks,
Jason

On Fri, Aug 25, 2017 at 10:48 AM, Mayuresh Gharat <
gharatmayures...@gmail.com> wrote:

> Hi Jason,
>
> Thanks a lot for the KIP and sorry for the delayed response.
>
> I had a few questions :
>
>
>- The KIP says that a user can have a class that extends KafkaPrincipal.
>Would this extended class be used when constructing the Session object
> in
>the SocketServer instead of constructing a new KafkaPrincipal?
>
>
>- The KIP says "A principal is always identifiable by a principal type
>and a name. Nothing else should ever be required." This might not be
> true
>always, right? For example, we might have a custom third party ACL
> library
>that creates a custom Principal from the passed in cert (this is done in
>PrincipalBuilder/KafkaPrincipalBuilder) and the custom Authorizer might
>use this third party library to authorize using this custom Principal
>object. The developer who is implementing the Kafka Authorizer should
>not be caring about what the custom Principal would look like and its
>details, since it will just pass it to the third party library in Kafka
>Authorizer's authorize() call.
>
>
> Thanks,
>
> Mayuresh
>
>
> On Thu, Aug 24, 2017 at 10:21 AM, Mayuresh Gharat <
> gharatmayures...@gmail.com> wrote:
>
> > Sure.
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Wed, Aug 23, 2017 at 5:07 PM, Jun Rao  wrote:
> >
> >> Hi, Mayuresh,
> >>
> >> Since this KIP covers the requirement in KIP-111, could you review it
> too?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Tue, Aug 22, 2017 at 3:04 PM, Jason Gustafson 
> >> wrote:
> >>
> >>> Bump. I'll open a vote in a few days if there are no comments.
> >>>
> >>> Thanks,
> >>> Jason
> >>>
> >>> On Sat, Aug 19, 2017 at 12:28 AM, Ismael Juma 
> wrote:
> >>>
> >>> > Thanks for the KIP Jason. It seems reasonable and cleans up some
> >>> > inconsistencies in that area. It would be great to get some feedback
> >>> from
> >>> > Mayuresh and others who worked on KIP-111.
> >>> 

[GitHub] kafka pull request #3738: KAFKA-5790: SocketServer.processNewResponses shoul...

[ijuma]

Github user ijuma closed the pull request at:

https://github.com/apache/kafka/pull/3738


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk8 #1944

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Consolidate broker request/response handling

--
[...truncated 2.46 MB...]

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldNotAllowNullTopicOnTo PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldAllowNullStoreInJoin STARTED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldAllowNullStoreInJoin PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > testRepartition 
STARTED

org.apache.kafka.streams.kstream.internals.KTableImplTest > testRepartition 
PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
testStateStoreLazyEval STARTED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
testStateStoreLazyEval PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldNotAllowNullStoreSupplierInJoin STARTED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldNotAllowNullStoreSupplierInJoin PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldNotAllowNullStoreSupplierInLeftJoin STARTED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldNotAllowNullStoreSupplierInLeftJoin PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldNotAllowNullActionOnForEach STARTED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldNotAllowNullActionOnForEach PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > testKTable STARTED

org.apache.kafka.streams.kstream.internals.KTableImplTest > testKTable PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldNotAllowNullPredicateOnFilterNot STARTED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldNotAllowNullPredicateOnFilterNot PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldAllowNullStoreInThrough STARTED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldAllowNullStoreInThrough PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > testValueGetter 
STARTED

org.apache.kafka.streams.kstream.internals.KTableImplTest > testValueGetter 
PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldNotAllowNullFilePathOnWriteAsText STARTED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldNotAllowNullFilePathOnWriteAsText PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldAllowNullTopicInThrough STARTED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
shouldAllowNullTopicInThrough PASSED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldRemoveMergedSessionsFromStateStore STARTED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldRemoveMergedSessionsFromStateStore PASSED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldMergeSessions STARTED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldMergeSessions PASSED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldHandleMultipleSessionsAndMerging STARTED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldHandleMultipleSessionsAndMerging PASSED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldImmediatelyForwardNewSessionWhenNonCachedStore STARTED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldImmediatelyForwardNewSessionWhenNonCachedStore PASSED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldGetAggregatedValuesFromValueGetter STARTED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldGetAggregatedValuesFromValueGetter PASSED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldImmediatelyForwardRemovedSessionsWhenMerging STARTED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldImmediatelyForwardRemovedSessionsWhenMerging PASSED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldUpdateSessionIfTheSameTime STARTED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldUpdateSessionIfTheSameTime PASSED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldHaveMultipleSessionsForSameIdWhenTimestampApartBySessionGap STARTED

org.apache.kafka.streams.kstream.internals.KStreamSessionWindowAggregateProcessorTest
 > shouldHaveMultipleSessionsForSameIdWhenTimestampApartBySessionGap PASSED

Jenkins build is back to normal : kafka-trunk-jdk7 #2677

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-5791) Add HTTPS support to the fault injector

[Colin P. McCabe (JIRA)]

Colin P. McCabe created KAFKA-5791:
--

 Summary: Add HTTPS support to the fault injector
 Key: KAFKA-5791
 URL: https://issues.apache.org/jira/browse/KAFKA-5791
 Project: Kafka
  Issue Type: Sub-task
Reporter: Colin P. McCabe


Add HTTPS support to the fault injector



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #2104: [KAFKA-4380] Remove cleanshutdownfile

[holdenk]

Github user holdenk closed the pull request at:

https://github.com/apache/kafka/pull/2104


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3742: [KAFKA-4380] Document the purpose of clean shutdow...

[holdenk]

GitHub user holdenk opened a pull request:

https://github.com/apache/kafka/pull/3742

[KAFKA-4380] Document the purpose of clean shutdown file in the code.

Remove the previous TODO to remove the clean shutdown file with some of the 
discussion from https://github.com/apache/kafka/pull/2104.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/holdenk/kafka 
KAFKA-4380-document-clean-shutdown-file

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3742.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3742


commit c43bc54b3195fdac575255e59f0e6e62dcb89dba
Author: Holden Karau 
Date:   2017-08-25T18:03:08Z

Replace the todo clearer description of why cleaning it up probably isn't a 
great idea




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

[Mayuresh Gharat]

Hi Jason,

Thanks a lot for the KIP and sorry for the delayed response.

I had a few questions :


   - The KIP says that a user can have a class that extends KafkaPrincipal.
   Would this extended class be used when constructing the Session object in
   the SocketServer instead of constructing a new KafkaPrincipal?


   - The KIP says "A principal is always identifiable by a principal type
   and a name. Nothing else should ever be required." This might not be true
   always, right? For example, we might have a custom third party ACL library
   that creates a custom Principal from the passed in cert (this is done in
   PrincipalBuilder/KafkaPrincipalBuilder) and the custom Authorizer might
   use this third party library to authorize using this custom Principal
   object. The developer who is implementing the Kafka Authorizer should
   not be caring about what the custom Principal would look like and its
   details, since it will just pass it to the third party library in Kafka
   Authorizer's authorize() call.


Thanks,

Mayuresh


On Thu, Aug 24, 2017 at 10:21 AM, Mayuresh Gharat <
gharatmayures...@gmail.com> wrote:

> Sure.
>
> Thanks,
>
> Mayuresh
>
> On Wed, Aug 23, 2017 at 5:07 PM, Jun Rao  wrote:
>
>> Hi, Mayuresh,
>>
>> Since this KIP covers the requirement in KIP-111, could you review it too?
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Tue, Aug 22, 2017 at 3:04 PM, Jason Gustafson 
>> wrote:
>>
>>> Bump. I'll open a vote in a few days if there are no comments.
>>>
>>> Thanks,
>>> Jason
>>>
>>> On Sat, Aug 19, 2017 at 12:28 AM, Ismael Juma  wrote:
>>>
>>> > Thanks for the KIP Jason. It seems reasonable and cleans up some
>>> > inconsistencies in that area. It would be great to get some feedback
>>> from
>>> > Mayuresh and others who worked on KIP-111.
>>> >
>>> > Ismael
>>> >
>>> > On Thu, Aug 17, 2017 at 1:21 AM, Jason Gustafson 
>>> > wrote:
>>> >
>>> > > Hi All,
>>> > >
>>> > > I've added a new KIP to improve and extend the principal building API
>>> > that
>>> > > Kafka exposes:
>>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> > > 189%3A+Improve+principal+builder+interface+and+add+support+for+SASL
>>> > > .
>>> > >
>>> > > As always, feedback is appreciated.
>>> > >
>>> > > Thanks,
>>> > > Jason
>>> > >
>>> >
>>>
>>
>>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>



-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

[jira] [Resolved] (KAFKA-4787) KafkaStreams close() is not reentrant

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-4787.
--
Resolution: Duplicate

> KafkaStreams close() is not reentrant
> -
>
> Key: KAFKA-4787
> URL: https://issues.apache.org/jira/browse/KAFKA-4787
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Steven Schlansker
>
> While building a simple application, I tried to implement a failure policy 
> where any uncaught exception terminates the application until an 
> administrator can evaluate and intervene:
> {code}
> /** Handle any uncaught exception by shutting down the program. */
> private void handleStreamException(Thread thread, Throwable t) {
> LOG.error("stream exception in thread {}", thread, t);
> streams.close();
> }
> streams.setUncaughtExceptionHandler(this::handleStreamException);
> streams.start();
> {code}
> Unfortunately, because the KafkaStreams#close() method takes a lock, this is 
> prone to what looks like a deadlock:
> {code}
> "StreamThread-1" #80 prio=5 os_prio=0 tid=0x7f56096f4000 nid=0x40c8 
> waiting for monitor entry [0x7f54f03ee000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java)
> - waiting to lock <0xf171cda8> (a 
> org.apache.kafka.streams.KafkaStreams)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:438)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$161/1940967023.close(Unknown 
> Source)
> at 
> com.opentable.chat.service.ChatStorage.closeLog(ChatStorage.java:212)
> at com.opentable.chat.service.ChatStorage.close(ChatStorage.java:207)
> at 
> com.opentable.chat.service.ChatStorage.handleStreamException(ChatStorage.java:541)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$123/149062221.uncaughtException(Unknown
>  Source)
> at java.lang.Thread.dispatchUncaughtException(Thread.java:1956)
> "main" #1 prio=5 os_prio=0 tid=0x7f5608011000 nid=0x3f76 in Object.wait() 
> [0x7f5610f04000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1249)
> - locked <0xfd302bf0> (a java.lang.Thread)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:494)
> - locked <0xf171cda8> (a 
> org.apache.kafka.streams.KafkaStreams)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:438)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$161/1940967023.close(Unknown 
> Source)
> at 
> com.opentable.chat.service.ChatStorage.closeLog(ChatStorage.java:212)
> at com.opentable.chat.service.ChatStorage.close(ChatStorage.java:207)
> {code}
> Note how the main thread calls close(), which encounters an exception.  It 
> uses a StreamThread to dispatch to the handler, which calls close().  Once it 
> tries to take the monitor, we are left in a position where main is joined on 
> StreamThread-1, but StreamThread-1 is waiting for main to release that 
> monitor.
> Arguably it's a bit abusive to call close() in this way (it certainly wasn't 
> intentional) -- but to make Kafka Streams robust it should handle any 
> sequence of close() invocations in particular gracefully.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3673: MINOR: Consolidate broker request/response handlin...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3673


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-3826) Sampling on throughput / latency metrics recording in Streams

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-3826.
--
Resolution: Fixed

This is been done as part of KAFKA-5152.

> Sampling on throughput / latency metrics recording in Streams
> -
>
> Key: KAFKA-3826
> URL: https://issues.apache.org/jira/browse/KAFKA-3826
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>  Labels: architecture, performance
>
> In Kafka Streams we record throughput / latency metrics on EACH processing 
> record, causing a lot of recording overhead. Instead, we should consider 
> statistically sampling messages flowing through to measures latency and 
> throughput.
> This is based on our observations from KAFKA-3769 and KAFKA-3811.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

[Pranav Maniar]

Yes I can take it up deprecation of  "log.cleaner.enable"

Will it require KIP ?
Since as per my understanding we will be honoring value set for
"log.cleaner.enable" till the time it is around. For now just a warning
message about deprecation will be logged only.

Or should we remove the cofig now only?


Thanks,
Pranav

On Wed, Aug 23, 2017 at 3:37 AM, Jason Gustafson  wrote:

> Hi Pranav,
>
> Yeah, I'd recommend closing it since the benefit is unclear and since no
> one has jumped in to offer stronger support for the change. Were you
> planning to do a KIP to deprecate `log.cleaner.enable`? I still think that
> makes sense.
>
> Thanks,
> Jason
>
> On Tue, Aug 22, 2017 at 1:47 PM, Colin McCabe  wrote:
>
> > Hmm.  There are a lot of configuration keys that involve "log cleaner."
> > It seems like if we rename this component, logically we'd have to rename
> > all of them and support the old versions as deprecated config keys:
> >
> >   val LogCleanupPolicyProp = "log.cleanup.policy"
> >   val LogCleanerThreadsProp = "log.cleaner.threads"
> >   val LogCleanerIoMaxBytesPerSecondProp =
> >   "log.cleaner.io.max.bytes.per.second"
> >   val LogCleanerDedupeBufferSizeProp = "log.cleaner.dedupe.buffer.size"
> >   val LogCleanerIoBufferSizeProp = "log.cleaner.io.buffer.size"
> >   val LogCleanerDedupeBufferLoadFactorProp =
> >   "log.cleaner.io.buffer.load.factor"
> >   val LogCleanerBackoffMsProp = "log.cleaner.backoff.ms"
> >   val LogCleanerMinCleanRatioProp = "log.cleaner.min.cleanable.ratio"
> >   val LogCleanerEnableProp = "log.cleaner.enable"
> >   val LogCleanerDeleteRetentionMsProp =
> >   "log.cleaner.delete.retention.ms"
> >   val LogCleanerMinCompactionLagMsProp =
> >   "log.cleaner.min.compaction.lag.ms"
> >
> > This seems like it would be quite painful to users, since they'd have to
> > deal with deprecation warnings and multiple names for the same
> > configuration.  In general I think Jason and Ismael's point is valid: do
> > we have evidence that "log cleaner" is causing confusion?  If not, it
> > may not be worth it to rename this at the moment.
> >
> > regards,
> > Colin
> >
> >
> > On Mon, Aug 21, 2017, at 05:19, Pranav Maniar wrote:
> > > Hi Jason,
> > >
> > > Haven't heard from other on this KIP. Should I close it ?
> > >
> > > ~Pranav
> > >
> > > On Thu, Aug 10, 2017 at 12:04 AM, Jason Gustafson 
> > > wrote:
> > >
> > > > Hey Pranav,
> > > >
> > > > Let's see what others think before closing the KIP. If there are
> strong
> > > > reasons for the renaming, I would reconsider.
> > > >
> > > > As far as deprecating `log.cleaner.enable`, I think it's a good idea
> > and
> > > > can be done in a separate KIP. Guozhang's suggestion seems
> reasonable,
> > but
> > > > I'd just turn it on always (it won't cause much harm if there are no
> > topics
> > > > enabled for compaction). This is an implementation detail which
> > probably
> > > > doesn't need to be included in the KIP.
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Aug 9, 2017 at 10:47 AM, Pranav Maniar  >
> > > > wrote:
> > > >
> > > > > Thanks Ismael, Jason for the suggestion.
> > > > > My bad. I should have followed up on mail-list discussion before
> > starting
> > > > > KIP. Apologies.
> > > > >
> > > > > I am relatively new, so I do not know if any confusion was reported
> > in
> > > > past
> > > > > due to terminology. May be others can chime in.
> > > > > If the old naming is fine with majority then no changes will be
> > needed. I
> > > > > will mark JIRA as wont'fix and close the KIP !
> > > > >
> > > > > Ismael, Jason,
> > > > > There was another suggestion from Guozhang on deprecating and
> > eventually
> > > > > removing log.cleaner.enable property all together and always
> > enabling log
> > > > > cleaner if "log.cleanup.policy=compact".
> > > > > What are your suggestion on this ?
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Pranav
> > > > >
> > > > > On Wed, Aug 9, 2017 at 10:27 PM, Jason Gustafson <
> ja...@confluent.io
> > >
> > > > > wrote:
> > > > >
> > > > > > Yes, as Ismael noted above, I am not fond of this renaming. Keep
> in
> > > > mind
> > > > > > that the LogCleaner does not only handle compaction. It is
> > possible to
> > > > > > configure a cleanup policy of "compact" and "delete," in which
> > case the
> > > > > > LogCleaner also handles removal of old segments. Hence the more
> > general
> > > > > > LogCleaner name is more appropriate in my opinion.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, Aug 9, 2017 at 9:49 AM, Pranav Maniar <
> > pranav9...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Ewen for the suggestions.
> > > > > > > I have updated KIP-184. Updates done are :
> > > > > > >
> > > > > > > 1. If deprecated property is encountered currently, then its
> > value
> > > > will
> > > > > > be
> > > > > > > considered while enabling compactor.
> > > > > > > 2.  

[jira] [Resolved] (KAFKA-1944) Rename LogCleaner and related classes to LogCompactor

[Pranav Maniar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pranav Maniar resolved KAFKA-1944.
--
Resolution: Won't Fix

As discussed on Mailing List and KIP-184 discussion. 
This name changes will lead to many configuration name changes. Which seems to 
be unnecessary trouble for users. 
Also there are no evidence of users getting confused because of the naming. 
So it is decided that we will stick with the existing names

> Rename LogCleaner and related classes to LogCompactor
> -
>
> Key: KAFKA-1944
> URL: https://issues.apache.org/jira/browse/KAFKA-1944
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Pranav Maniar
>  Labels: newbie
>
> Following a mailing list discussion:
> "the name LogCleaner is seriously misleading. Its more of a log compactor. 
> Deleting old logs happens elsewhere from what I've seen."
> Note that this may require renaming related classes, objects, configs and 
> metrics.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

[Jun Rao]

Hi, Rajini,

Thanks for the KIP. +1

Jun

On Thu, Aug 24, 2017 at 10:29 AM, Rajini Sivaram 
wrote:

> Hi all,
>
> I would like to start vote on KIP-152 to improve diagnostics of
> authentication failures and to update clients to treat authentication
> failures as fatal exceptions rather than transient errors:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 152+-+Improve+diagnostics+for+SASL+authentication+failures
>
> Thank you...
>
> Rajini
>

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

[Jun Rao]

Hi, Becket,

Good point on expiring inflight requests. Perhaps we can expire an inflight
request after min(remaining delivery.timeout.ms, request.timeout.ms). This
way, if a user sets a high delivery.timeout.ms, we can still recover from
broker power outage sooner.

Thanks,

Jun

On Thu, Aug 24, 2017 at 12:52 PM, Becket Qin  wrote:

> Hi Jason,
>
> delivery.timeout.ms sounds good to me.
>
> I was referring to the case that we are resetting the PID/sequence after
> expire a batch. This is more about the sending the batches after the
> expired batch.
>
> The scenario being discussed is expiring one of the batches in a in-flight
> request and retry the other batches in the that in-flight request. So
> consider the following case:
> 1. Producer sends request_0 with two batches (batch_0_tp0 and batch_0_tp1).
> 2. Broker receives the request enqueued the request to the log.
> 3. Before the producer receives the response from the broker, batch_0_tp0
> expires. The producer will expire batch_0_tp0 immediately, resets PID, and
> then resend batch_0_tp1, and maybe send batch_1_tp0 (i.e. the next batch to
> the expired batch) as well.
>
> For batch_0_tp1, it is OK to reuse PID and and sequence number. The problem
> is for batch_1_tp0, If we reuse the same PID and the broker has already
> appended batch_0_tp0, the broker will think batch_1_tp0 is a duplicate with
> the same sequence number. As a result broker will drop batch_0_tp1. That is
> why we have to either bump up sequence number or reset PID. To avoid this
> complexity, I was suggesting not expire the in-flight batch immediately,
> but wait for the produce response. If the batch has been successfully
> appended, we do not expire it. Otherwise, we expire it.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
> On Thu, Aug 24, 2017 at 11:26 AM, Jason Gustafson 
> wrote:
>
> > @Becket
> >
> > Good point about unnecessarily resetting the PID in cases where we know
> the
> > request has failed. Might be worth opening a JIRA to try and improve
> this.
> >
> > So if we expire the batch prematurely and resend all
> > > the other batches in the same request, chances are there will be
> > > duplicates. If we wait for the response instead, it is less likely to
> > > introduce duplicates, and we may not need to reset the PID.
> >
> >
> > Not sure I follow this. Are you assuming that we change the batch
> > PID/sequence of the retried batches after resetting the PID? I think we
> > probably need to ensure that when we retry a batch, we always use the
> same
> > PID/sequence.
> >
> > By the way, as far as naming, `max.message.delivery.wait.ms` is quite a
> > mouthful. Could we shorten it? Perhaps `delivery.timeout.ms`?
> >
> > -Jason
> >
> > On Wed, Aug 23, 2017 at 8:51 PM, Becket Qin 
> wrote:
> >
> > > Hi Jun,
> > >
> > > If TCP timeout is longer than request.timeout.ms, the producer will
> > always
> > > hit request.timeout.ms before hitting TCP timeout, right? That is why
> we
> > > added request.timeout.ms in the first place.
> > >
> > > You are right. Currently we are reset the PID and resend the batches to
> > > avoid OutOfOrderSequenceException when the expired batches are in
> retry.
> > >
> > > This does not distinguish the reasons that caused the retry. There are
> > two
> > > cases:
> > > 1. If the batch was in retry because it received an error response
> (e.g.
> > > NotLeaderForPartition), we actually don't need to reset PID in this
> case
> > > because we know that broker did not accept it.
> > > 2. If the batch was in retry because it hit a timeout earlier, then we
> > > should reset the PID (or optimistically send and only reset PID when
> > > receive OutOfOrderSequenceException?)
> > > Case 1 is probably the most common case, so it looks that we are
> > resetting
> > > the PID more often than necessary. But because in case 1 the broker
> does
> > > not have the batch, there isn't much impact on resting PID and resend
> > other
> > > than the additional round trip.
> > >
> > > Now we are introducing another case:
> > > 3. A batch is in retry because we expired an in-flight request before
> it
> > > hits request.timeout.ms.
> > >
> > > The difference between 2 and 3 is that in case 3 likely the broker has
> > > appended the messages. So if we expire the batch prematurely and resend
> > all
> > > the other batches in the same request, chances are there will be
> > > duplicates. If we wait for the response instead, it is less likely to
> > > introduce duplicates, and we may not need to reset the PID.
> > >
> > > That said, given that batch expiration is probably already rare enough,
> > so
> > > it may not be necessary to optimize for that.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > >
> > >
> > > On Wed, Aug 23, 2017 at 5:01 PM, Jun Rao  wrote:
> > >
> > > > Hi, Becket,
> > > >
> > > > If a message expires while it's in an inflight produce request, the
> > > > 

Re: [VOTE] KIP-182 - Reduce Streams DSL overloads and allow easier use of custom storage engines

[Matthias J. Sax]

Thanks Damian. Great KIP!

+1


-Matthias

On 8/25/17 6:45 AM, Damian Guy wrote:
> Hi,
> 
> I've just realised we need to add two methods to StateStoreBuilder or it
> isn't going to work:
> 
> Map logConfig();
> boolean loggingEnabled();
> 
> These are needed when we are building the topology and determining
> changelog topic names and configs.
> 
> 
> I've also update the KIP to add
> 
> StreamBuilder#stream(String topic)
> 
> StreamBuilder#stream(String topic, Consumed options)
> 
> 
> Thanks
> 
> 
> On Thu, 24 Aug 2017 at 22:11 Sriram Subramanian  wrote:
> 
>> +1
>>
>> On Thu, Aug 24, 2017 at 10:20 AM, Guozhang Wang 
>> wrote:
>>
>>> +1. Thanks Damian!
>>>
>>> On Thu, Aug 24, 2017 at 9:47 AM, Bill Bejeck  wrote:
>>>
 Thanks for the KIP!

 +1

 Thanks,
 Bill

 On Thu, Aug 24, 2017 at 12:25 PM, Damian Guy 
>>> wrote:

> Hi,
>
> I'd like to kick off the voting thread for KIP-182:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 182%3A+Reduce+Streams+DSL+overloads+and+allow+easier+
> use+of+custom+storage+engines
>
> Thanks,
> Damian
>

>>>
>>>
>>>
>>> --
>>> -- Guozhang
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Jenkins build is back to normal : kafka-trunk-jdk8 #1943

[Apache Jenkins Server]

See 

[GitHub] kafka pull request #3741: KAFKA-5762; LogContext used to capture the clientI...

[Kamal15]

GitHub user Kamal15 opened a pull request:

https://github.com/apache/kafka/pull/3741

KAFKA-5762; LogContext used to capture the clientId implicitly in the…

… logs.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Kamal15/kafka kafka-5762

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3741.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3741


commit 274a9688a004f63b74a8997f84082456e45fe785
Author: Kamal Chandraprakash 
Date:   2017-08-25T14:46:57Z

KAFKA-5762; LogContext used to capture the clientId implicitly in the logs.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #2676

[Apache Jenkins Server]

See 

--
[...truncated 915.80 KB...]

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica 
STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldSurviveFastLeaderChange STARTED

Re: [VOTE] KIP-182 - Reduce Streams DSL overloads and allow easier use of custom storage engines

[Damian Guy]

Hi,

I've just realised we need to add two methods to StateStoreBuilder or it
isn't going to work:

Map logConfig();
boolean loggingEnabled();

These are needed when we are building the topology and determining
changelog topic names and configs.


I've also update the KIP to add

StreamBuilder#stream(String topic)

StreamBuilder#stream(String topic, Consumed options)


Thanks


On Thu, 24 Aug 2017 at 22:11 Sriram Subramanian  wrote:

> +1
>
> On Thu, Aug 24, 2017 at 10:20 AM, Guozhang Wang 
> wrote:
>
> > +1. Thanks Damian!
> >
> > On Thu, Aug 24, 2017 at 9:47 AM, Bill Bejeck  wrote:
> >
> > > Thanks for the KIP!
> > >
> > > +1
> > >
> > > Thanks,
> > > Bill
> > >
> > > On Thu, Aug 24, 2017 at 12:25 PM, Damian Guy 
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I'd like to kick off the voting thread for KIP-182:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 182%3A+Reduce+Streams+DSL+overloads+and+allow+easier+
> > > > use+of+custom+storage+engines
> > > >
> > > > Thanks,
> > > > Damian
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Build failed in Jenkins: kafka-trunk-jdk7 #2675

[Apache Jenkins Server]

See 


Changes:

[damian.guy] KAFKA-5771; 
org.apache.kafka.streams.state.internals.Segments#segments

--
[...truncated 916.05 KB...]

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica 
STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown PASSED

[jira] [Created] (KAFKA-5790) SocketServer.processNewResponses should not skip a response if exception is thrown

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-5790:
--

 Summary: SocketServer.processNewResponses should not skip a 
response if exception is thrown
 Key: KAFKA-5790
 URL: https://issues.apache.org/jira/browse/KAFKA-5790
 Project: Kafka
  Issue Type: Bug
Reporter: Ismael Juma
Assignee: Ismael Juma
Priority: Critical
 Fix For: 0.11.0.1, 1.0.0


The current code has a try/finally that causes a response to be dropped when an 
exception is thrown during `processNewResponses`. This could one of the reasons 
for KAFKA-4669.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3737: KAFKA-5771: org.apache.kafka.streams.state.interna...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3737


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-5771) org.apache.kafka.streams.state.internals.Segments#segments method returns incorrect results when segments were added out of order

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy resolved KAFKA-5771.
---
   Resolution: Fixed
Fix Version/s: 0.11.0.1
   1.0.0

Issue resolved by pull request 3737
[https://github.com/apache/kafka/pull/3737]

> org.apache.kafka.streams.state.internals.Segments#segments method returns 
> incorrect results when segments were added out of order
> -
>
> Key: KAFKA-5771
> URL: https://issues.apache.org/jira/browse/KAFKA-5771
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Alexander Radzishevsky
> Fix For: 1.0.0, 0.11.0.1
>
>
> following unit test in org.apache.kafka.streams.state.internals.SegmentsTest 
> will fail
> {code:title=org.apache.kafka.streams.state.internals.SegmentsTest.java|borderStyle=solid}
> @Test
> public void shouldGetSegmentsWithinTimeRangeOutOfOrder() throws Exception 
> {
> segments.getOrCreateSegment(4, context);
> segments.getOrCreateSegment(2, context);
> segments.getOrCreateSegment(0, context);
> segments.getOrCreateSegment(1, context);
> segments.getOrCreateSegment(3, context);
> final List segments = this.segments.segments(0, 2 * 60 * 
> 1000);
> assertEquals(3, segments.size());
> assertEquals(0, segments.get(0).id);
> assertEquals(1, segments.get(1).id);
> assertEquals(2, segments.get(2).id);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3465: MINOR: Fixed the invalid link to Log Compaction

[Kamal15]

Github user Kamal15 closed the pull request at:

https://github.com/apache/kafka/pull/3465


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3740: MINOR: reduce logging to trace in NetworkClient wh...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3740

MINOR: reduce logging to trace in NetworkClient when an old server API is 
being used

logging in `NetworkClient#doSend´ at debug level is spamming the logs when 
you have a producer that is sending many requests. It makes it extremely 
difficult to debug. Reducing to trace to remove the noise from the logs

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka minor-network-client-logging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3740.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3740


commit 89a07a5bbdd11bd2e88bd98dae9bde48d3530eaf
Author: Damian Guy 
Date:   2017-08-25T11:25:36Z

reduce logging to trace in NetworkClient when an old server API is being 
used




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-4869) 0.10.2.0 release notes incorrectly include KIP-115

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4869.
--
Resolution: Fixed
  Assignee: Manikumar

> 0.10.2.0 release notes incorrectly include KIP-115
> --
>
> Key: KAFKA-4869
> URL: https://issues.apache.org/jira/browse/KAFKA-4869
> Project: Kafka
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 0.10.2.0
>Reporter: Yeva Byzek
>Assignee: Manikumar
>Priority: Minor
>
> From http://kafka.apache.org/documentation.html :
> bq. The offsets.topic.replication.factor broker config is now enforced upon 
> auto topic creation. Internal auto topic creation will fail with a 
> GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets this 
> replication factor requirement.
> Even though this feature 
> [KIP-115|https://cwiki.apache.org/confluence/display/KAFKA/KIP-115%3A+Enforce+offsets.topic.replication.factor+upon+__consumer_offsets+auto+topic+creation]
>  did not make it into 0.10.2.0



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3739: KAFKA-5789: Deleted topic is recreated when consum...

[stakafum]

GitHub user stakafum opened a pull request:

https://github.com/apache/kafka/pull/3739

KAFKA-5789: Deleted topic is recreated when consumer subscribe the deleted 
one



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/stakafum/kafka KAFKA-5789

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3739.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3739


commit 82a51f999cececc9c3a0dcc99231592b1efdf8dc
Author: saito-takafumi 
Date:   2017-08-25T10:24:16Z

KAFKA-5789: disable auto topic creation by consumer




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3735: KAFKA-4869: Update 0.10.2.0 upgrade notes

[omkreddy]

Github user omkreddy closed the pull request at:

https://github.com/apache/kafka/pull/3735


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-5789) Deleted topic is recreated when consumer subscribe the deleted one

[Takafumi Saito (JIRA)]

Takafumi Saito created KAFKA-5789:
-

 Summary: Deleted topic is recreated when consumer subscribe the 
deleted one
 Key: KAFKA-5789
 URL: https://issues.apache.org/jira/browse/KAFKA-5789
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.11.0.0
Reporter: Takafumi Saito


When setting auto.create.topic.enbale true in broker, some deleted topics will 
be re-created.
Because when consumer that subscribe deleted topic is exist, broker will create 
topic having same name.
It is not necessary that consumers trigger new topic creation , so auto topic 
creation in consumer should be disabled.

I attatch the log outputted in our broker.
This show that a topic (topic_1) was deleted at 12:02:24,672, but same topic 
was created shortly thereafter:

{code:java}
[2017-08-22 12:02:24,666] INFO [ReplicaFetcherManager on broker 1] Removed 
fetcher for partitions topic_1-0 (kafka.server.ReplicaFetcherManager)
[2017-08-22 12:02:24,666] INFO [ReplicaFetcherManager on broker 1] Removed 
fetcher for partitions  (kafka.server.ReplicaFetcherManager)
[2017-08-22 12:02:24,667] INFO [ReplicaFetcherManager on broker 1] Removed 
fetcher for partitions topic_1-0 (kafka.server.ReplicaFetcherManager)
[2017-08-22 12:02:24,672] INFO Log for partition topic_1-0 is renamed to 
/data/topic_1-0.ad490e8326704ae6a6fd9f6399c29614-delete and is scheduled for 
deletion (kafka.log.LogManager)
[2017-08-22 12:02:24,736] INFO Loading producer state from offset 0 for 
partition topic_1-0 with message format version 2 (kafka.log.Log)
[2017-08-22 12:02:24,736] INFO Completed load of log topic_1-0 with 1 log 
segments, log start offset 0 and log end offset 0 in 1 ms (kafka.log.Log)
[2017-08-22 12:02:24,737] INFO Created log for partition [topic_1,0] in /data 
with properties {compression.type -> producer, message.format.version -> 
0.11.0-IV2, file.delete.delay.ms -> 6,
max.message.bytes -> 112, min.compaction.lag.ms -> 0, 
message.timestamp.type -> CreateTime, min.insync.replicas -> 1, 
segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, 
i
ndex.interval.bytes -> 4096, unclean.leader.election.enable -> false, 
retention.bytes -> -1, delete.retention.ms -> 8640, cleanup.policy -> 
[delete], flush.ms -> 9223372036854775807, segment.ms -> 60
480, segment.bytes -> 1073741824, retention.ms -> 8640, 
message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes 
-> 10485760, flush.messages -> 9223372036854775807}. (kafka
.log.LogManager)
[2017-08-22 12:02:24,738] INFO [ReplicaFetcherManager on broker 1] Removed 
fetcher for partitions topic_1-0 (kafka.server.ReplicaFetcherManager)
[2017-08-22 12:02:24,738] INFO [ReplicaFetcherManager on broker 1] Added 
fetcher for partitions List([topic_1-0, initOffset 0 to broker 
BrokerEndPoint(2,sbx-patriot-kafka02.amb-patriot.incvb.io,
9092)] ) (kafka.server.ReplicaFetcherManager)
[2017-08-22 12:02:25,200] INFO [ReplicaFetcherThread-0-2]: Based on follower's 
leader epoch, leader replied with an offset 0 >= the follower's log end offset 
0 in topic_1-0. No truncation needed
. (kafka.server.ReplicaFetcherThread)
[2017-08-22 12:02:25,200] INFO Truncating topic_1-0 to 0 has no effect as the 
largest offset in the log is -1. (kafka.log.Log)
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3738: MINOR: Remove try/finally from processNewResponses

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/3738

MINOR: Remove try/finally from processNewResponses

The previous code would skip a response if an exception was thrown.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka fix-process-new-responses

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3738.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3738


commit 68175a15c327ab9f31e9e2120601310694e1c57a
Author: Ismael Juma 
Date:   2017-08-25T10:16:03Z

MINOR: Remove try/finally from processNewResponses




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-5788) "IllegalArgumentException: long is not a value type" when running ReassignPartitionsCommand

[Ansel Zandegran (JIRA)]

Ansel Zandegran created KAFKA-5788:
--

 Summary: "IllegalArgumentException: long is not a value type" when 
running ReassignPartitionsCommand
 Key: KAFKA-5788
 URL: https://issues.apache.org/jira/browse/KAFKA-5788
 Project: Kafka
  Issue Type: Bug
  Components: admin
Affects Versions: 0.11.0.0
 Environment: Windows 
Reporter: Ansel Zandegran


When trying to run ReassignPartitionsCommand with the following commands.
String[] reassignCmdArgs = { "--reassignment-json-file=" + 
Paths.get(reassignmentConfigFileName),
"--zookeeper=" + 
client.getZookeeperClient().getCurrentConnectionString(), "--execute", 
"--throttle="+1000 };
logger.debug("Calling 
ReassignPartitionsCommand with args:{}", Arrays.toString(reassignCmdArgs));

ReassignPartitionsCommand.main(reassignCmdArgs);


2017-08-22 15:57:28 DEBUG ZookeeperBackedAdoptionLogicImpl:320 - Calling 
ReassignPartitionsCommand with 
args:[--reassignment-json-file=partitions-to-move.json.1503417447767, 
--zookeeper=172.31.14.207:2181, --execute]
java.lang.IllegalArgumentException: long is not a value type
at joptsimple.internal.Reflection.findConverter(Reflection.java:66)
at 
joptsimple.ArgumentAcceptingOptionSpec.ofType(ArgumentAcceptingOptionSpec.java:111)
at 
kafka.admin.ReassignPartitionsCommand$ReassignPartitionsCommandOptions.(ReassignPartitionsCommand.scala:301)
at 
kafka.admin.ReassignPartitionsCommand$.validateAndParseArgs(ReassignPartitionsCommand.scala:236)
at 
kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:34)
at 
kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
at 
rebalancer.core.ZookeeperBackedAdoptionLogicImpl.reassignPartitionToLocalBroker(ZookeeperBackedAdoptionLogicImpl.java:321)
at 
rebalancer.core.ZookeeperBackedAdoptionLogicImpl.adoptRemotePartition(ZookeeperBackedAdoptionLogicImpl.java:267)
at 
rebalancer.core.ZookeeperBackedAdoptionLogicImpl.run(ZookeeperBackedAdoptionLogicImpl.java:118)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3737: KAFKA-5771: org.apache.kafka.streams.state.interna...

[radzish]

GitHub user radzish opened a pull request:

https://github.com/apache/kafka/pull/3737

KAFKA-5771: org.apache.kafka.streams.state.internals.Segments#segments 
method returns incorrect results when segments were added out of order

Suggested fix for the bug

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/radzish/kafka KAFKA-5771

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3737.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3737


commit 32f8da5d0b39c449303a8dcd34476ed2b68197e6
Author: radzish 
Date:   2017-08-25T09:39:18Z

KAFKA-5771: org.apache.kafka.streams.state.internals.Segments#segments 
method returns incorrect results when segments were added out of order




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3736: KAFKA-5787: StoreChangelogReader needs to restore ...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3736

KAFKA-5787: StoreChangelogReader needs to restore partitions that were 
added post initialization

If a task fails during initialization due to a LockException, its changelog 
partitions are not immediately added to the StoreChangelogReader as the thread 
doesn't hold the lock. However StoreChangelogReader#restore will be called and 
it sets the initialized flag. On a subsequent successfull call to initialize 
the new tasks the partitions are added to the StoreChangelogReader, however as 
it is already initialized these new partitions will never be restored. So the 
task would remain in a non-running state forever.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-5787

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3736.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3736


commit 451b6a746cc5173b22d8110497bbbe33842d18a3
Author: Damian Guy 
Date:   2017-08-25T09:36:05Z

StoreChangelogReader needs to restore partitions that were added post 
initialization




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3735: KAFKA-4869: Update 0.10.2.0 upgrade notes

[omkreddy]

GitHub user omkreddy opened a pull request:

https://github.com/apache/kafka/pull/3735

KAFKA-4869: Update 0.10.2.0 upgrade notes



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omkreddy/kafka cleanup10

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3735.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3735


commit 1409ca459054a4e661863410419f346375e2be22
Author: Manikumar Reddy 
Date:   2017-08-25T09:17:54Z

KAFKA-4869: Update 0.10.2.0 upgrade notes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-5787) StoreChangeLogReader needs to restore partitions that were added post initialization

[Damian Guy (JIRA)]

Damian Guy created KAFKA-5787:
-

 Summary: StoreChangeLogReader needs to restore partitions that 
were added post initialization
 Key: KAFKA-5787
 URL: https://issues.apache.org/jira/browse/KAFKA-5787
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.11.0.1, 1.0.0
Reporter: Damian Guy
Assignee: Damian Guy
Priority: Blocker


Investigation of {{KStreamRepartitionJoinTest}} failures uncovered this bug. If 
a task fails during initialization due to a {{LockException}}, its changelog 
partitions are not immediately added to the {{StoreChangelogReader}} as the 
thread doesn't hold the lock. However {{StoreChangelogReader#restore}} will be 
called and it sets the initialized flag. On a subsequent successfull call to 
initialize the new tasks the partitions are added to the 
{{StoreChangelogReader}}, however as it is already initialized these new 
partitions will never be restored. So the task will remain in a non-running 
state forever



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3734: KAFKA-5785; Always close connection if KafkaChanne...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/3734

KAFKA-5785; Always close connection if KafkaChannel.setSend throws exception



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-5785-always-close-connection-if-set-send-throws-exception

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3734.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3734


commit 12d2fb4c151b2786b9f6db687cae6cda55155f37
Author: Ismael Juma 
Date:   2017-08-25T09:06:32Z

KAFKA-5785; Always close connection if KafkaChannel.setSend throws exception




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

[Mickael Maison]

Huge +1 !
Clients getting no feedback when failing SASL authentication is a
major source of confusion. Thanks for the KIP

On Fri, Aug 25, 2017 at 9:48 AM, Manikumar  wrote:
> +1 (non-binding)
>
>
> On Fri, Aug 25, 2017 at 12:37 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
>> +1
>>
>> Thanks Rajini.
>>
>> --Vahid
>>
>>
>>
>> From:   Edoardo Comar 
>> To: dev@kafka.apache.org
>> Date:   08/24/2017 10:55 AM
>> Subject:Re: [VOTE] KIP-152 - Improve diagnostics for SASL
>> authentication failures
>>
>>
>>
>> Thanks Rajini!
>>
>> +1 (non-binding)
>> --
>>
>> Edoardo Comar
>>
>> IBM Message Hub
>>
>> IBM UK Ltd, Hursley Park, SO21 2JN
>>
>>
>>
>> From:   Rajini Sivaram 
>> To: dev 
>> Date:   24/08/2017 18:30
>> Subject:[VOTE] KIP-152 - Improve diagnostics for SASL
>> authentication failures
>>
>>
>>
>> Hi all,
>>
>> I would like to start vote on KIP-152 to improve diagnostics of
>> authentication failures and to update clients to treat authentication
>> failures as fatal exceptions rather than transient errors:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
>> apache.org_confluence_display_KAFKA_KIP-2D152-2B-2D-
>> 2BImprove-2Bdiagnostics-2Bfor-2BSASL-2Bauthentication-
>> 2Bfailures=DwIBAg=jf_iaSHvJObTbx-siA1ZOg=Q_
>> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=oomurFvX7GYTMP0ZlfXU8eQSS68Mfe
>> CQYhvtB7zXrCw=AaHp43w6RWNlh_1HZc6_rdangxC-oXMWHz7XR6i0suA=
>>
>>
>>
>> Thank you...
>>
>> Rajini
>>
>>
>>
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number
>> 741598.
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>
>>
>>
>>
>>

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

[Manikumar]

+1 (non-binding)


On Fri, Aug 25, 2017 at 12:37 AM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> +1
>
> Thanks Rajini.
>
> --Vahid
>
>
>
> From:   Edoardo Comar 
> To: dev@kafka.apache.org
> Date:   08/24/2017 10:55 AM
> Subject:Re: [VOTE] KIP-152 - Improve diagnostics for SASL
> authentication failures
>
>
>
> Thanks Rajini!
>
> +1 (non-binding)
> --
>
> Edoardo Comar
>
> IBM Message Hub
>
> IBM UK Ltd, Hursley Park, SO21 2JN
>
>
>
> From:   Rajini Sivaram 
> To: dev 
> Date:   24/08/2017 18:30
> Subject:[VOTE] KIP-152 - Improve diagnostics for SASL
> authentication failures
>
>
>
> Hi all,
>
> I would like to start vote on KIP-152 to improve diagnostics of
> authentication failures and to update clients to treat authentication
> failures as fatal exceptions rather than transient errors:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> apache.org_confluence_display_KAFKA_KIP-2D152-2B-2D-
> 2BImprove-2Bdiagnostics-2Bfor-2BSASL-2Bauthentication-
> 2Bfailures=DwIBAg=jf_iaSHvJObTbx-siA1ZOg=Q_
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=oomurFvX7GYTMP0ZlfXU8eQSS68Mfe
> CQYhvtB7zXrCw=AaHp43w6RWNlh_1HZc6_rdangxC-oXMWHz7XR6i0suA=
>
>
>
> Thank you...
>
> Rajini
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
>
>
>

[jira] [Created] (KAFKA-5786) Yet another exception is causing that streamming app is zombie

[Seweryn Habdank-Wojewodzki (JIRA)]

Seweryn Habdank-Wojewodzki created KAFKA-5786:
-

 Summary: Yet another exception is causing that streamming app is 
zombie
 Key: KAFKA-5786
 URL: https://issues.apache.org/jira/browse/KAFKA-5786
 Project: Kafka
  Issue Type: Bug
Reporter: Seweryn Habdank-Wojewodzki
Priority: Critical


Not handled exception in streamming app causes zombie state of the process.

{code}
2017-08-24 15:17:40 WARN  StreamThread:978 - stream-thread 
[kafka-endpoint-1236e6d5-75f0-4c14-b025-78e632484a26-StreamThread-3] Unexpected 
state transition from RUNNING to DEAD.
2017-08-24 15:17:40 FATAL StreamProcessor:67 - Caught unhandled exception: 
stream-thread 
[kafka-endpoint-1236e6d5-75f0-4c14-b025-78e632484a26-StreamThread-3] Failed to 
rebalance.; 
[org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:589),
 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553),
 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)]
 in thread kafka-endpoint-1236e6d5-75f0-4c14-b025-78e632484a26-StreamThread-3
{code}

The final state of the app is similar to KAFKA-5779, but the exception and its 
location is in different place.

The exception shall be handled in the way that either application tries to 
continue working or shall completely quit if the error is not recoverable.

Current situation when application is zombie is not good.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5785) Always close connection if KafkaChannel.setSend throws exception

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-5785:
--

 Summary: Always close connection if KafkaChannel.setSend throws 
exception
 Key: KAFKA-5785
 URL: https://issues.apache.org/jira/browse/KAFKA-5785
 Project: Kafka
  Issue Type: Bug
Reporter: Ismael Juma
Assignee: Ismael Juma
 Fix For: 0.11.0.1, 1.0.0


The code is currently:

{code}
try {
channel.setSend(send);
} catch (CancelledKeyException e) {
this.failedSends.add(connectionId);
close(channel, false);
}
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-trunk-jdk8 #1942

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: KafkaService should print node hostname on failure

--
[...truncated 2.46 MB...]

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testNoMessagesSentExceptionFromOverlappingPatterns STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testNoMessagesSentExceptionFromOverlappingPatterns PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
shouldAddStateStoreToRegexDefinedSource STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
shouldAddStateStoreToRegexDefinedSource PASSED

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperationsWithZeroSizedCache STARTED

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperationsWithZeroSizedCache FAILED
java.lang.AssertionError: Condition not met within timeout 6. Expecting 
5 records from topic map-one-join-output-1 while only received 0: []
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:275)
at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinValuesRecordsReceived(IntegrationTestUtils.java:201)
at 
org.apache.kafka.streams.integration.KStreamRepartitionJoinTest.receiveMessages(KStreamRepartitionJoinTest.java:375)
at 
org.apache.kafka.streams.integration.KStreamRepartitionJoinTest.verifyCorrectOutput(KStreamRepartitionJoinTest.java:296)
at 
org.apache.kafka.streams.integration.KStreamRepartitionJoinTest.verifyRepartitionOnJoinOperations(KStreamRepartitionJoinTest.java:141)
at 
org.apache.kafka.streams.integration.KStreamRepartitionJoinTest.shouldCorrectlyRepartitionOnJoinOperationsWithZeroSizedCache(KStreamRepartitionJoinTest.java:119)

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperationsWithNonZeroSizedCache STARTED

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperationsWithNonZeroSizedCache PASSED

org.apache.kafka.streams.integration.InternalTopicIntegrationTest > 
shouldCompactTopicsForStateChangelogs STARTED

org.apache.kafka.streams.integration.InternalTopicIntegrationTest > 
shouldCompactTopicsForStateChangelogs PASSED

org.apache.kafka.streams.integration.InternalTopicIntegrationTest > 
shouldUseCompactAndDeleteForWindowStoreChangelogs STARTED

org.apache.kafka.streams.integration.InternalTopicIntegrationTest > 
shouldUseCompactAndDeleteForWindowStoreChangelogs PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduceSessionWindows STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduceSessionWindows PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduce STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduce PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldAggregate STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldAggregate PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCount STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCount PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldGroupByKey STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldGroupByKey PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCountWithInternalStore STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCountWithInternalStore PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduceWindowed STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduceWindowed PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCountSessionWindows STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCountSessionWindows PASSED

[jira] [Resolved] (KAFKA-5595) Illegal state in SocketServer; attempt to send with another send in progress

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-5595.

Resolution: Fixed

I'm going to close this. If we see the issue again, we can reopen it.

> Illegal state in SocketServer; attempt to send with another send in progress
> 
>
> Key: KAFKA-5595
> URL: https://issues.apache.org/jira/browse/KAFKA-5595
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Rajini Sivaram
> Fix For: 1.0.0
>
>
> I have seen this a couple times, but I'm not sure the conditions associated 
> with it. 
> {code}
> java.lang.IllegalStateException: Attempt to begin a send operation with prior 
> send operation still in progress.
>   at 
> org.apache.kafka.common.network.KafkaChannel.setSend(KafkaChannel.java:138)
>   at org.apache.kafka.common.network.Selector.send(Selector.java:248)
>   at kafka.network.Processor.sendResponse(SocketServer.scala:488)
>   at kafka.network.Processor.processNewResponses(SocketServer.scala:466)
>   at kafka.network.Processor.run(SocketServer.scala:431)
>   at java.lang.Thread.run(Thread.java:748)
> {code}
> Prior to this event, I see a lot of this message in the logs (always for the 
> same connection id):
> {code}
> Attempting to send response via channel for which there is no open 
> connection, connection id 7
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (KAFKA-5595) Illegal state in SocketServer; attempt to send with another send in progress

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reopened KAFKA-5595:


> Illegal state in SocketServer; attempt to send with another send in progress
> 
>
> Key: KAFKA-5595
> URL: https://issues.apache.org/jira/browse/KAFKA-5595
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Rajini Sivaram
> Fix For: 1.0.0
>
>
> I have seen this a couple times, but I'm not sure the conditions associated 
> with it. 
> {code}
> java.lang.IllegalStateException: Attempt to begin a send operation with prior 
> send operation still in progress.
>   at 
> org.apache.kafka.common.network.KafkaChannel.setSend(KafkaChannel.java:138)
>   at org.apache.kafka.common.network.Selector.send(Selector.java:248)
>   at kafka.network.Processor.sendResponse(SocketServer.scala:488)
>   at kafka.network.Processor.processNewResponses(SocketServer.scala:466)
>   at kafka.network.Processor.run(SocketServer.scala:431)
>   at java.lang.Thread.run(Thread.java:748)
> {code}
> Prior to this event, I see a lot of this message in the logs (always for the 
> same connection id):
> {code}
> Attempting to send response via channel for which there is no open 
> connection, connection id 7
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5595) Illegal state in SocketServer; attempt to send with another send in progress

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-5595.

   Resolution: Fixed
 Assignee: Rajini Sivaram
Fix Version/s: 1.0.0

> Illegal state in SocketServer; attempt to send with another send in progress
> 
>
> Key: KAFKA-5595
> URL: https://issues.apache.org/jira/browse/KAFKA-5595
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Rajini Sivaram
> Fix For: 1.0.0
>
>
> I have seen this a couple times, but I'm not sure the conditions associated 
> with it. 
> {code}
> java.lang.IllegalStateException: Attempt to begin a send operation with prior 
> send operation still in progress.
>   at 
> org.apache.kafka.common.network.KafkaChannel.setSend(KafkaChannel.java:138)
>   at org.apache.kafka.common.network.Selector.send(Selector.java:248)
>   at kafka.network.Processor.sendResponse(SocketServer.scala:488)
>   at kafka.network.Processor.processNewResponses(SocketServer.scala:466)
>   at kafka.network.Processor.run(SocketServer.scala:431)
>   at java.lang.Thread.run(Thread.java:748)
> {code}
> Prior to this event, I see a lot of this message in the logs (always for the 
> same connection id):
> {code}
> Attempting to send response via channel for which there is no open 
> connection, connection id 7
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

[Damian Guy]

On Thu, 24 Aug 2017 at 18:31 Xavier Léauté  wrote:

> A few comments on the KIP:
> - I'm a bit confused about the BytesStoreSupplier interface. Nothing in its
> definition is really specific to Bytes, and
> when I see return types like BytesStoreSupplier byte[]>>, it seems redundant to have "Bytes" in the supplier name.
> Why can't we reuse the existing StateStoreSupplier interface and move the
> extra logConfig and loggingEnabled methods elsewhere?
>

We can't re-use StateStoreSupplier as it would break compatibility. So we
needed another name.


> - I don't really see any mention of the motivation behind the Materialized
> interface and what the implications are for the user, i.e. what does it
> mean for a store to be materialized.
>

It means that there will be a store either created for you with the store
name provided or with the BytesStoreSupplier. It provides a convenient way
for users to enable/disable caching and logging on a per store basis. And
helps to reduce the current overloads spread throughout the code.


> - Until now, serialization implementation details were decoupled from the
> state store interfaces. With this KIP we are now bubbling up the
> assumptions that state store going to be using Bytes or byte[] into the
> API.

I'm not a fan of this, because it precludes us from providing more
> efficient implementations, e.g. using ByteBuffers, that can avoid costly
> byte array copying and extraneous byte array allocations during serde
> operations.
> A better approach might be to provide a first class ByteStore interface
> that could help abstract the different types of buffers we might want to
> use, or alternatively use a buffer agnostic type in the state store
> definition (similar to what LMDB
> <
> https://github.com/lmdbjava/lmdbjava/blob/master/src/main/java/org/lmdbjava/BufferProxy.java
> >
>  does)
>
>
We decided to do it this way as we want to provide developers with the
ability to use our wrapper stores, i.e, ChangeLogging, Caching, Metered.
Presently they can't do this without jumping through various hoops.
To provide this ability we presently need to use  as the
CachingStores are . They need to remain that way for the
time being as it is how we can put some limits on memory usage.


> On Thu, Aug 24, 2017 at 1:53 AM Damian Guy  wrote:
>
> > I've updated the kip to reflect Bill's comment and also to make
> > StreamBuilder methods have topic as the first param, i.e.,
> > StreamBuilder#stream no longer accepts varargs.
> >
> > On Thu, 24 Aug 2017 at 09:12 Damian Guy  wrote:
> >
> > > On Thu, 24 Aug 2017 at 02:49 Guozhang Wang  wrote:
> > >
> > >> I have a couple of comments but otherwise it LGTM:
> > >>
> > >> 1. For these two functions in StreamsBuilder, the topic String is set
> as
> > >> the second parameter in between of two options. Would that be better
> to
> > be
> > >> set as the first or the last one instead?
> > >>
> > >> It would be better as the first, but then it is different to the
> > > #streams() methods due to varargs.
> > >
> > >
> > >> public synchronized  KTable table(final Consumed
> > >> consumed, final String topic, final Materialized materialized)
> > >>
> > >> public synchronized  GlobalKTable globalTable(final
> > >> Consumed > >> V> consumed, final String topic, final Materialized
> materialized)
> > >>
> > >> I understand that we cannot do it for the first parameter because of
> the
> > >> vararg type. So I'd suggest either
> > >>
> > >> a) set it as the last parameter, but then it is inconsistent with
> other
> > >> functions like these:
> > >>
> > >> void to(final String topic, final Produced options);
> > >>
> > >> KTable through(final String topic, final Materialized
> > >> options);
> > >>
> > >> b) only allow one single topic name parameter in
> StreamsBuilder.stream()
> > >> since in practice we do not see too many usages of multiple topics,
> plus
> > >> it
> > >> can be semi-supported with "merge" as we move it from StreamsBuilder
> to
> > >> KStream (KAFKA-5765),
> > >>
> > >> Perhaps this is the better approach
> > >
> > >
> > >> 2. KGroupedStream's function:
> > >>
> > >>  KTable aggregate(final Initializer initializer,
> > >>  final Aggregator VR>
> > >> aggregator,
> > >>  final Serde aggValueSerde,
> > >>  final Materialized KeyValueStore > >> VR>> materialized);
> > >>
> > >> The "aggValueSerde" seems not needed?
> > >>
> > >> 3. +1 on `KGroupedStream` v.s. `GroupedKStream`. I think
> KGroupedStream
> > >> was
> > >> a bad name as a hind-sight. I personally feel we should just correct
> it
> > >> with a new class and deprecate / remove the old one before 1.0.0, but
> > that
> > >> could be in its own KIP.
> > >>
> > >>
> > > The 

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

[Damian Guy]

Matthias, i agree so i've added those two overloads.
Thanks,
Damian

On Thu, 24 Aug 2017 at 21:54 Matthias J. Sax  wrote:

> Thanks for clarification. I see your point. Java varargs are problematic
> in general IMHO as they force you to put them as last argument making
> parameter ordering unnatural for some cases (as we have it currently in
> the API).
>
> Nevertheless, I think that reading a single topic is the most common
> case and thus I would love to see the overloads as mentioned in my last
> email in addition to the overloads taking a Collection of topics. Maybe
> it's just personal taste -- I agree that the overhead of specifying a
> singleton on not severe, but to me it still feels like a "step backward"
> as reading a single topic should be the pattern for like 90% or more of
> the cases.
>
>
> -Matthias
>
>
> On 8/24/17 12:03 PM, Guozhang Wang wrote:
> > Matthias,
> >
> > I think it's my bad that I did not post another comment on the mailing
> list
> > while syncing with Damian. Here it is:
> >
> > Regarding 1) above, a second thought on varargs: though I have not heard
> > from anyone using multiple topics, it is also true that people will just
> > keep silent until their APIs gets removed. So instead of keeping a single
> > topic name in the constructor, it'd better to still allow users to pass
> > multiple topics, as a Collection topic.
> >
> > It does mean that users who would only want a single topic would feel
> > inconvenient with "Collections.singleton(topic)", but I felt it is not
> too
> > big of an issue. On the other hand KafkaConsumer also only allow
> > `subscribe(Collection topics)` so I'd suggest in this KIP we do
> not
> > have two overloads of "stream(topic)" and "stream(topics)" and consider
> > adding that as a syntax-sugar if it does become a big complaint.
> >
> >
> > Guozhang
> >
> >
> >
> > On Thu, Aug 24, 2017 at 11:32 AM, Matthias J. Sax  >
> > wrote:
> >
> >> We now have
> >>
> >>> public synchronized  KStream stream(final
> Collection
> >> topic, final Consumed options)
> >>
> >> This would prevent so write code like
> >>
> >> builder.stream("topic", Consumers.with(...));
> >>
> >> I think, we need methods
> >>
> >> StreamsBuilder#stream(String topic);
> >> StreamsBuilder#stream(String topic, Consumed options);
> >>
> >> Or do I miss anything?
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 8/24/17 1:53 AM, Damian Guy wrote:
> >>> I've updated the kip to reflect Bill's comment and also to make
> >>> StreamBuilder methods have topic as the first param, i.e.,
> >>> StreamBuilder#stream no longer accepts varargs.
> >>>
> >>> On Thu, 24 Aug 2017 at 09:12 Damian Guy  wrote:
> >>>
>  On Thu, 24 Aug 2017 at 02:49 Guozhang Wang 
> wrote:
> 
> > I have a couple of comments but otherwise it LGTM:
> >
> > 1. For these two functions in StreamsBuilder, the topic String is set
> >> as
> > the second parameter in between of two options. Would that be better
> >> to be
> > set as the first or the last one instead?
> >
> > It would be better as the first, but then it is different to the
>  #streams() methods due to varargs.
> 
> 
> > public synchronized  KTable table(final Consumed
> > consumed, final String topic, final Materialized materialized)
> >
> > public synchronized  GlobalKTable globalTable(final
> > Consumed > V> consumed, final String topic, final Materialized
> materialized)
> >
> > I understand that we cannot do it for the first parameter because of
> >> the
> > vararg type. So I'd suggest either
> >
> > a) set it as the last parameter, but then it is inconsistent with
> other
> > functions like these:
> >
> > void to(final String topic, final Produced options);
> >
> > KTable through(final String topic, final Materialized
> > options);
> >
> > b) only allow one single topic name parameter in
> >> StreamsBuilder.stream()
> > since in practice we do not see too many usages of multiple topics,
> >> plus
> > it
> > can be semi-supported with "merge" as we move it from StreamsBuilder
> to
> > KStream (KAFKA-5765),
> >
> > Perhaps this is the better approach
> 
> 
> > 2. KGroupedStream's function:
> >
> >  KTable aggregate(final Initializer initializer,
> >  final Aggregator VR>
> > aggregator,
> >  final Serde aggValueSerde,
> >  final Materialized KeyValueStore > VR>> materialized);
> >
> > The "aggValueSerde" seems not needed?
> >
> > 3. +1 on `KGroupedStream` v.s. `GroupedKStream`. I think
> KGroupedStream
> > was
> > a bad name as a hind-sight. I personally feel we should just correct
> it
> 

Re: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

[James Cheng]

Congrats Becket!

-James

> On Aug 23, 2017, at 10:20 PM, Joel Koshy  wrote:
> 
> Hi everyone,
> 
> Jiangjie (Becket) Qin has been a Kafka committer in October 2016 and has
> contributed significantly to several major patches, reviews and discussions
> since. I am glad to announce that Becket is now a member of the Apache Kafka
> PMC.
> 
> Congratulations Becket!
> 
> Joel

Jenkins build is back to normal : kafka-trunk-jdk8 #1941

[Apache Jenkins Server]

See 

[GitHub] kafka pull request #3715: KafkaService: print node hostname on failure

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3715


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka-site pull request #71: KAFKA-4869: Update 0.10.2.0 upgrade notes

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka-site/pull/71


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka-site issue #71: KAFKA-4869: Update 0.10.2.0 upgrade notes

[ijuma]

Github user ijuma commented on the issue:

https://github.com/apache/kafka-site/pull/71
  
Thanks for the PR, we also need to fix it in the main repo: 
https://github.com/apache/kafka/blob/0.10.2/docs/upgrade.html#L90


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---