Re: [VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

[Apurva Mehta]

Yes, you are right. It is worth mentioning since the default value of the
retries config is changing.

Thanks,
Apurva

On Thu, Sep 7, 2017 at 10:25 PM, Sumant Tambe  wrote:

> I added the following: Due to change in the default value of retries from 0
> to MAX_INT and the existing default value of
> max.in.flight.request.per.connection==5, reordering becomes a possibility
> by default. To prevent reordering, set
> max.in.flight.request.per.connection==1.
>
> It does not hurt to mention it as it's a default behavior change?
>
> On 7 September 2017 at 22:15, Apurva Mehta  wrote:
>
> > Thanks for the KIP Sumant, +1 from me.
> >
> > That is the most exhaustive 'Rejected Alternatives' section that I have
> > seen :)
> >
> > One minor point: In the compatibility section, your note on
> > 'max.in.flight.requests.per.connection == 5' resulting in out of order
> > delivery is true irrespective of these changes. As such, I don't think it
> > should be mentioned in the context of this KIP.
> >
> > Thanks,
> > Apurva
> >
> > On Thu, Sep 7, 2017 at 10:06 PM, Sumant Tambe  wrote:
> >
> > > Hi all,
> > >
> > > I would like to open the vote for KIP-91:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 91+Provide+Intuitive+User+Timeouts+in+The+Producer
> > >
> > > Thank you all for your input on the kip so far.
> > >
> > > Regards,
> > > Sumant
> > >
> >
>

Re: [VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

[Sumant Tambe]

I added the following: Due to change in the default value of retries from 0
to MAX_INT and the existing default value of
max.in.flight.request.per.connection==5, reordering becomes a possibility
by default. To prevent reordering, set
max.in.flight.request.per.connection==1.

It does not hurt to mention it as it's a default behavior change?

On 7 September 2017 at 22:15, Apurva Mehta  wrote:

> Thanks for the KIP Sumant, +1 from me.
>
> That is the most exhaustive 'Rejected Alternatives' section that I have
> seen :)
>
> One minor point: In the compatibility section, your note on
> 'max.in.flight.requests.per.connection == 5' resulting in out of order
> delivery is true irrespective of these changes. As such, I don't think it
> should be mentioned in the context of this KIP.
>
> Thanks,
> Apurva
>
> On Thu, Sep 7, 2017 at 10:06 PM, Sumant Tambe  wrote:
>
> > Hi all,
> >
> > I would like to open the vote for KIP-91:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 91+Provide+Intuitive+User+Timeouts+in+The+Producer
> >
> > Thank you all for your input on the kip so far.
> >
> > Regards,
> > Sumant
> >
>

Re: [VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

[Apurva Mehta]

Thanks for the KIP Sumant, +1 from me.

That is the most exhaustive 'Rejected Alternatives' section that I have
seen :)

One minor point: In the compatibility section, your note on
'max.in.flight.requests.per.connection == 5' resulting in out of order
delivery is true irrespective of these changes. As such, I don't think it
should be mentioned in the context of this KIP.

Thanks,
Apurva

On Thu, Sep 7, 2017 at 10:06 PM, Sumant Tambe  wrote:

> Hi all,
>
> I would like to open the vote for KIP-91:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 91+Provide+Intuitive+User+Timeouts+in+The+Producer
>
> Thank you all for your input on the kip so far.
>
> Regards,
> Sumant
>

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

[Sumant Tambe]

Just did :)

On 7 September 2017 at 17:52, Ismael Juma  wrote:

> Can we please start the vote on this KIP? The KIP must be accepted by next
> Wednesday in order to make the cut for 1.0.0. This issue keeps coming up
> again and again, and I'd really like to include a fix for 1.0.0.
>
> Ismael
>
> On Thu, Sep 7, 2017 at 10:01 PM, Apurva Mehta  wrote:
>
> > I agree with what Ismael said. Having both retries and
> delivery.timeout.ms
> > is confusing, and thus the goal is to not have a retries option at all
> once
> > idempotence is fully battle tested and has become the entrenched default.
> >
> > Until that time, it makes sense to expire batch earlier than
> > delivery.timeout.ms if retries have been exhausted.
> >
> > Thanks,
> > Apurva
> >
> >
> > On Thu, Sep 7, 2017 at 6:07 AM, Ismael Juma  wrote:
> >
> > > Good question regarding retries Sumant. A few comments:
> > >
> > > 1. Defaulting to MAX_INT makes sense in the context of
> > delivery.timeout.ms
> > > ,
> > > but introduces the possibility of reordering with the default
> > max.in.flight
> > > of 5. Personally, I think reordering is better than dropping the
> message
> > > altogether (if we keep retries=0), but it's worth noting this.
> > >
> > > 2. I agree that we should expire on whichever of retries and
> > > delivery.timeout.ms is exhausted first for 1.0.0.
> > >
> > > 3. Once KIP-185 lands (post 1.0.0), we should consider deprecating and
> > > eventually removing the retries config to simplify things (it won't
> have
> > > much use then).
> > >
> > > 4. With regards to the case where the broker replies quickly with an
> > error,
> > > we need to understand a bit more what the error is. For any kind of
> > > connection issue, we now have exponential backoff. For the case where
> an
> > > error code is returned, it depends on whether the error is retriable or
> > > not. For the former, it probably makes sense to keep retrying as it's
> > > supposed to be a transient issue. If we think it would make sense to
> > apply
> > > exponential backoff, we could also consider that. So, I'm not sure
> > retries
> > > has much use apart from compatibility and the retries=0 case (for now).
> > >
> > > Ismael
> > >
> > > On Wed, Sep 6, 2017 at 11:14 PM, Jun Rao  wrote:
> > >
> > > > Hi, Sumant,
> > > >
> > > > The diagram in the wiki seems to imply that delivery.timeout.ms
> > doesn't
> > > > include the batching time.
> > > >
> > > > For retries, probably we can just default it to MAX_INT?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Wed, Sep 6, 2017 at 10:26 AM, Sumant Tambe 
> > wrote:
> > > >
> > > > > 120 seconds default sounds good to me. Throwing ConfigException
> > instead
> > > > of
> > > > > WARN is fine. Added clarification that the producer waits the full
> > > > > request.timeout.ms for the in-flight request. This implies that
> user
> > > > might
> > > > > be notified of batch expiry while a batch is still in-flight.
> > > > >
> > > > > I don't recall if we discussed our point of view that existing
> > configs
> > > > like
> > > > > retries become redundant/deprecated with this feature. IMO, retries
> > > > config
> > > > > becomes meaningless due to the possibility of incorrect configs
> like
> > > > > delivery.timeout.ms > linger.ms + retries * (request..timeout.ms +
> > > > > retry.backoff.ms), retries should be basically interpreted as
> > MAX_INT?
> > > > > What
> > > > > will be the default?
> > > > >
> > > > > So do we ignore retries config or throw a ConfigException if
> > weirdness
> > > > like
> > > > > above is detected?
> > > > >
> > > > > -Sumant
> > > > >
> > > > >
> > > > > On 5 September 2017 at 17:34, Ismael Juma 
> wrote:
> > > > >
> > > > > > Thanks for updating the KIP, Sumant. A couple of points:
> > > > > >
> > > > > > 1. I think the default for delivery.timeout.ms should be higher
> > than
> > > > 30
> > > > > > seconds given that we previously would reset the clock once the
> > batch
> > > > was
> > > > > > sent. The value should be large enough that batches are not
> expired
> > > due
> > > > > to
> > > > > > expected events like a new leader being elected due to broker
> > > failure.
> > > > > > Would it make sense to use a conservative value like 120 seconds?
> > > > > >
> > > > > > 2. The producer currently throws an exception for configuration
> > > > > > combinations that don't make sense. We should probably do the
> same
> > > here
> > > > > for
> > > > > > consistency (the KIP currently proposes a log warning).
> > > > > >
> > > > > > 3. We should mention that we will not cancel in flight requests
> > until
> > > > the
> > > > > > request timeout even though we'll expire the batch early if
> needed.
> > > > > >
> > > > > > I think we should start the vote tomorrow so that we have a
> chance
> > of
> > > > > > hitting the KIP freeze for 1.0.0.
> > > > > >
> > > > > > Ismael

[VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

[Sumant Tambe]

Hi all,

I would like to open the vote for KIP-91:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-91+Provide+Intuitive+User+Timeouts+in+The+Producer

Thank you all for your input on the kip so far.

Regards,
Sumant

Build failed in Jenkins: kafka-trunk-jdk8 #1995

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Include response in request log

--
[...truncated 2.51 MB...]

org.apache.kafka.streams.integration.GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableJoin PASSED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithIntermediateUserTopic STARTED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithIntermediateUserTopic PASSED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithoutIntermediateUserTopic STARTED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithoutIntermediateUserTopic PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.KStreamAggregationDedupIntegrationTest > 
shouldReduce STARTED

org.apache.kafka.streams.integration.KStreamAggregationDedupIntegrationTest > 
shouldReduce PASSED

org.apache.kafka.streams.integration.KStreamAggregationDedupIntegrationTest > 
shouldGroupByKey STARTED

Re: [VOTE] KIP-192 : Provide cleaner semantics when idempotence is enabled

[Apurva Mehta]

Thanks for the comments Ismael.

I have gone ahead and incorporated all your suggestions in the KIP
document. You convinced me on adding max.message.bytes :)

Apurva

On Thu, Sep 7, 2017 at 6:12 PM, Ismael Juma  wrote:

> Thanks for the KIP. +1 (binding) from me. A few minor comments:
>
> 1. We should add a note to the backwards compatibility section explaining
> the impact of throwing DuplicateSequenceException (a new exception) from
> `send`. As I understand it, it's not an issue, but good to include it in
> the KIP.
>
> 2. For clarity, it's good to highlight in some way the new fields in the
> protocol definition itself
>
> 3. I understand that you decided not to add max.message.bytes because it's
> unrelated to this KIP. I'll try to persuade you that we should, but it's
> not a blocker if you don't agree. The reasons are: 1. The implementation
> effort to add it is minimal since it's a topic config like message format
> version, 2. It's clearly beneficial for the producer to have that
> information, 3. It's compact (just a number), 4. It's nice to avoid another
> protocol bump for a small change like that.
>
> Thanks,
> Ismael
>
> On Thu, Sep 7, 2017 at 9:51 PM, Apurva Mehta  wrote:
>
> > Hi,
> >
> > I'd like to start a vote for KIP-192:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
> >
> > Thanks,
> > Apurva
> >
>

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

[Randall Hauch]

The KIP and PR expose the OffsetStorageReader, which is already exposed to the 
tasks. The OffsetStorageWriter is part of the implementation, but was not and 
is not exposed thru the API. 

> On Sep 7, 2017, at 9:04 PM, Gwen Shapira  wrote:
> 
> I just re-read the code for the OffsetStorageWriter, and ran into this
> comment:
> 
> * Note that this only provides write functionality. This is
> intentional to ensure stale data is
> * never read. Offset data should only be read during startup or
> reconfiguration of a task. By
> * always serving those requests by reading the values from the backing
> store, we ensure we never
> * accidentally use stale data. (One example of how this can occur: a
> task is processing input
> * partition A, writing offsets; reconfiguration causes partition A to
> be reassigned elsewhere;
> * reconfiguration causes partition A to be reassigned to this node,
> but now the offset data is out
> * of date). Since these offsets are created and managed by the
> connector itself, there's no way
> * for the offset management layer to know which keys are "owned" by
> which tasks at any given
> * time.
> 
> 
> I can't figure out how the KIP avoids the stale-reads problem explained here.
> 
> Can you talk me through it? I'm cancelling my vote since right now
> exposing this interface sounds risky and misleading.
> 
> 
> Gwen
> 
> 
>> On Thu, Sep 7, 2017 at 5:04 PM Gwen Shapira  wrote:
>> 
>> +1 (binding)
>> 
>> Looking forward to see how connector implementations use this in practice
>> :)
>> 
>>> On Thu, Sep 7, 2017 at 3:49 PM Randall Hauch  wrote:
>>> 
>>> I'd like to open the vote for KIP-131:
>>> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector
>>> 
>>> Thanks to Florian for submitting the KIP and the implementation, and to
>>> everyone else that helped review.
>>> 
>>> Best regards,
>>> 
>>> Randall
>>> 
>> 

[GitHub] kafka pull request #3801: MINOR: Include response in request log

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3801


---

[GitHub] kafka pull request #3813: MINOR: Move request/response schemas to the corres...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/3813

MINOR: Move request/response schemas to the corresponding object 
representation

This refactor achieves the following:

1. Breaks up the increasingly unmanageable `Protocol` class.
2. Removes the need for redundant field identifiers maintained separately 
in `Protocol` and the respective request/response objects.
3. Provides a better mechanism for sharing common fields between different 
schemas (e.g. topics, partitions, error codes, etc.).
4. Adds convenience helpers to `Struct` for common patterns (such as 
setting a field only if it exists).


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka protocol-schema-refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3813.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3813


commit 9d008c806e43f7a13221fd5761aed79fb9b4f2cf
Author: Jason Gustafson 
Date:   2017-09-07T22:34:27Z

MINOR: Move request/response schemas to the corresponding object 
representation




---

Build failed in Jenkins: kafka-trunk-jdk8 #1994

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Always specify the keystore type in system tests

--
[...truncated 2.03 MB...]

org.apache.kafka.common.security.ssl.SslFactoryTest > testClientMode STARTED

org.apache.kafka.common.security.ssl.SslFactoryTest > testClientMode PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryConfiguration STARTED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryConfiguration PASSED

org.apache.kafka.common.security.kerberos.KerberosNameTest > testParse STARTED

org.apache.kafka.common.security.kerberos.KerberosNameTest > testParse PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testPrincipalNameCanContainSeparator STARTED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testPrincipalNameCanContainSeparator PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testEqualsAndHashCode STARTED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testEqualsAndHashCode PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameOverride STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameOverride PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingOptionValue 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingOptionValue PASSED

org.apache.kafka.common.security.JaasContextTest > testSingleOption STARTED

org.apache.kafka.common.security.JaasContextTest > testSingleOption PASSED

org.apache.kafka.common.security.JaasContextTest > 
testNumericOptionWithoutQuotes STARTED

org.apache.kafka.common.security.JaasContextTest > 
testNumericOptionWithoutQuotes PASSED

org.apache.kafka.common.security.JaasContextTest > testConfigNoOptions STARTED

org.apache.kafka.common.security.JaasContextTest > testConfigNoOptions PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithWrongListenerName STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithWrongListenerName PASSED

org.apache.kafka.common.security.JaasContextTest > testNumericOptionWithQuotes 
STARTED

org.apache.kafka.common.security.JaasContextTest > testNumericOptionWithQuotes 
PASSED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionValue STARTED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionValue PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingLoginModule 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingLoginModule PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingSemicolon STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingSemicolon PASSED

org.apache.kafka.common.security.JaasContextTest > testMultipleOptions STARTED

org.apache.kafka.common.security.JaasContextTest > testMultipleOptions PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForClientWithListenerName STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForClientWithListenerName PASSED

org.apache.kafka.common.security.JaasContextTest > testMultipleLoginModules 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMultipleLoginModules 
PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingControlFlag 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingControlFlag PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameAndFallback STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameAndFallback PASSED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionName STARTED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionName PASSED

org.apache.kafka.common.security.JaasContextTest > testControlFlag STARTED

org.apache.kafka.common.security.JaasContextTest > testControlFlag PASSED

org.apache.kafka.common.security.plain.PlainSaslServerTest > 
noAuthorizationIdSpecified STARTED

org.apache.kafka.common.security.plain.PlainSaslServerTest > 
noAuthorizationIdSpecified PASSED

org.apache.kafka.common.security.plain.PlainSaslServerTest > 
authorizatonIdEqualsAuthenticationId STARTED

org.apache.kafka.common.security.plain.PlainSaslServerTest > 
authorizatonIdEqualsAuthenticationId PASSED

org.apache.kafka.common.security.plain.PlainSaslServerTest > 
authorizatonIdNotEqualsAuthenticationId STARTED

org.apache.kafka.common.security.plain.PlainSaslServerTest > 
authorizatonIdNotEqualsAuthenticationId PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingUsernameSaslPlain STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingUsernameSaslPlain PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 

[jira] [Created] (KAFKA-5859) Avoid retaining AbstractRequest in RequestChannel.Request

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-5859:
--

 Summary: Avoid retaining AbstractRequest in RequestChannel.Request
 Key: KAFKA-5859
 URL: https://issues.apache.org/jira/browse/KAFKA-5859
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma
Priority: Minor


We currently store AbstractRequest in RequestChannel.Request.bodyAndSize. 
RequestChannel.Request is, in turn, stored in RequestChannel.Response. We keep 
the latter until the response is sent to the client.

However, after KafkaApis.handle, we no longer need AbstractRequest apart from 
its string representation for logging. We could potentially replace 
AbstractRequest with a String representation (if the relevant logging is 
enabled). The String representation is generally small while some 
AbstractRequest subclasses can be pretty large. The largest one is 
ProduceRequest and we clear the underlying ByteBuffer explicitly in 
KafkaApis.handleProduceRequest. We could potentially remove that special case 
if AbstractRequest subclasses were not retained.

This was originally suggested by [~hachikuji] in the following PR 
https://github.com/apache/kafka/pull/3801#discussion_r137592277



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

[Gwen Shapira]

Thanks for the KIP, Randall. Those are badly needed!

Can we have two metrics with record rate per task? One before SMT and one
after?
We can have cases where we read 5000 rows from JDBC but write 5 to Kafka,
or read 5000 records from Kafka and write 5 due to filtering. I think its
important to know both numbers.


Gwen

On Thu, Sep 7, 2017 at 7:50 PM, Randall Hauch  wrote:

> Hi everyone.
>
> I've created a new KIP to add metrics to the Kafka Connect framework:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 196%3A+Add+metrics+to+Kafka+Connect+framework
>
> The KIP approval deadline is looming, so if you're interested in Kafka
> Connect metrics please review and provide feedback as soon as possible. I'm
> interested not only in whether the metrics are sufficient and appropriate,
> but also in whether the MBean naming conventions are okay.
>
> Best regards,
>
> Randall
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter  | blog

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

[Gwen Shapira]

I just re-read the code for the OffsetStorageWriter, and ran into this
comment:

* Note that this only provides write functionality. This is
intentional to ensure stale data is
* never read. Offset data should only be read during startup or
reconfiguration of a task. By
* always serving those requests by reading the values from the backing
store, we ensure we never
* accidentally use stale data. (One example of how this can occur: a
task is processing input
* partition A, writing offsets; reconfiguration causes partition A to
be reassigned elsewhere;
* reconfiguration causes partition A to be reassigned to this node,
but now the offset data is out
* of date). Since these offsets are created and managed by the
connector itself, there's no way
* for the offset management layer to know which keys are "owned" by
which tasks at any given
* time.


I can't figure out how the KIP avoids the stale-reads problem explained here.

Can you talk me through it? I'm cancelling my vote since right now
exposing this interface sounds risky and misleading.


Gwen


On Thu, Sep 7, 2017 at 5:04 PM Gwen Shapira  wrote:

> +1 (binding)
>
> Looking forward to see how connector implementations use this in practice
> :)
>
> On Thu, Sep 7, 2017 at 3:49 PM Randall Hauch  wrote:
>
>> I'd like to open the vote for KIP-131:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector
>>
>> Thanks to Florian for submitting the KIP and the implementation, and to
>> everyone else that helped review.
>>
>> Best regards,
>>
>> Randall
>>
>

[jira] [Resolved] (KAFKA-5858) consumer.poll() shouldn't throw exception due to deserialization error

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-5858.

Resolution: Duplicate

Duplicate of KAFKA-5470,

> consumer.poll() shouldn't throw exception due to deserialization error
> --
>
> Key: KAFKA-5858
> URL: https://issues.apache.org/jira/browse/KAFKA-5858
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.11.0.0
>Reporter: Jun Rao
>
> Currently, the new consumer will throw an exception in poll() if it hits a 
> deserialization error. The consumer then can't make progress from this point 
> on. It will be better to throw the deserialization exception only when the 
> key/value of the ConsumerRecord is accessed, like the old consumer does.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3808: MINOR: Always specify the keystore type in system ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3808


---

Re: [VOTE] KIP-192 : Provide cleaner semantics when idempotence is enabled

[Ismael Juma]

Thanks for the KIP. +1 (binding) from me. A few minor comments:

1. We should add a note to the backwards compatibility section explaining
the impact of throwing DuplicateSequenceException (a new exception) from
`send`. As I understand it, it's not an issue, but good to include it in
the KIP.

2. For clarity, it's good to highlight in some way the new fields in the
protocol definition itself

3. I understand that you decided not to add max.message.bytes because it's
unrelated to this KIP. I'll try to persuade you that we should, but it's
not a blocker if you don't agree. The reasons are: 1. The implementation
effort to add it is minimal since it's a topic config like message format
version, 2. It's clearly beneficial for the producer to have that
information, 3. It's compact (just a number), 4. It's nice to avoid another
protocol bump for a small change like that.

Thanks,
Ismael

On Thu, Sep 7, 2017 at 9:51 PM, Apurva Mehta  wrote:

> Hi,
>
> I'd like to start a vote for KIP-192:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
>
> Thanks,
> Apurva
>

Re: [DISCUSS] KIP-195: AdminClient.increasePartitions

[Ismael Juma]

Thanks Tom. Thanks for the KIP. A few comments:

1. Does the `PartitionCount` class still make sense given that the method
can only increase the number of partitions now?
2,. In `NewTopic`, we have `numPartitions`. Should we keep using that
variant and call the method `increaseNumPartitions`?
3. Since this has been discussed at length as part of the reassign
partitions KIP, I suggest starting the vote tomorrow if there are no
objections from others.

Thanks,
Ismael

On Thu, Sep 7, 2017 at 5:38 PM, Tom Bentley  wrote:

> As suggested by Ismael, I've factored the increasePartitionCounts() API out
> of KIP-179 out into a separate KIP which hopefully can progress more
> quickly.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%
> 3A+AdminClient.increasePartitions
>
> If you've looked at KIP-179 in the last few days there's no much new to see
> here, but if not you should find KIP-195 a lighter read.
>
> Cheers,
>
> Tom
>

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

[Ismael Juma]

Can we please start the vote on this KIP? The KIP must be accepted by next
Wednesday in order to make the cut for 1.0.0. This issue keeps coming up
again and again, and I'd really like to include a fix for 1.0.0.

Ismael

On Thu, Sep 7, 2017 at 10:01 PM, Apurva Mehta  wrote:

> I agree with what Ismael said. Having both retries and delivery.timeout.ms
> is confusing, and thus the goal is to not have a retries option at all once
> idempotence is fully battle tested and has become the entrenched default.
>
> Until that time, it makes sense to expire batch earlier than
> delivery.timeout.ms if retries have been exhausted.
>
> Thanks,
> Apurva
>
>
> On Thu, Sep 7, 2017 at 6:07 AM, Ismael Juma  wrote:
>
> > Good question regarding retries Sumant. A few comments:
> >
> > 1. Defaulting to MAX_INT makes sense in the context of
> delivery.timeout.ms
> > ,
> > but introduces the possibility of reordering with the default
> max.in.flight
> > of 5. Personally, I think reordering is better than dropping the message
> > altogether (if we keep retries=0), but it's worth noting this.
> >
> > 2. I agree that we should expire on whichever of retries and
> > delivery.timeout.ms is exhausted first for 1.0.0.
> >
> > 3. Once KIP-185 lands (post 1.0.0), we should consider deprecating and
> > eventually removing the retries config to simplify things (it won't have
> > much use then).
> >
> > 4. With regards to the case where the broker replies quickly with an
> error,
> > we need to understand a bit more what the error is. For any kind of
> > connection issue, we now have exponential backoff. For the case where an
> > error code is returned, it depends on whether the error is retriable or
> > not. For the former, it probably makes sense to keep retrying as it's
> > supposed to be a transient issue. If we think it would make sense to
> apply
> > exponential backoff, we could also consider that. So, I'm not sure
> retries
> > has much use apart from compatibility and the retries=0 case (for now).
> >
> > Ismael
> >
> > On Wed, Sep 6, 2017 at 11:14 PM, Jun Rao  wrote:
> >
> > > Hi, Sumant,
> > >
> > > The diagram in the wiki seems to imply that delivery.timeout.ms
> doesn't
> > > include the batching time.
> > >
> > > For retries, probably we can just default it to MAX_INT?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, Sep 6, 2017 at 10:26 AM, Sumant Tambe 
> wrote:
> > >
> > > > 120 seconds default sounds good to me. Throwing ConfigException
> instead
> > > of
> > > > WARN is fine. Added clarification that the producer waits the full
> > > > request.timeout.ms for the in-flight request. This implies that user
> > > might
> > > > be notified of batch expiry while a batch is still in-flight.
> > > >
> > > > I don't recall if we discussed our point of view that existing
> configs
> > > like
> > > > retries become redundant/deprecated with this feature. IMO, retries
> > > config
> > > > becomes meaningless due to the possibility of incorrect configs like
> > > > delivery.timeout.ms > linger.ms + retries * (request..timeout.ms +
> > > > retry.backoff.ms), retries should be basically interpreted as
> MAX_INT?
> > > > What
> > > > will be the default?
> > > >
> > > > So do we ignore retries config or throw a ConfigException if
> weirdness
> > > like
> > > > above is detected?
> > > >
> > > > -Sumant
> > > >
> > > >
> > > > On 5 September 2017 at 17:34, Ismael Juma  wrote:
> > > >
> > > > > Thanks for updating the KIP, Sumant. A couple of points:
> > > > >
> > > > > 1. I think the default for delivery.timeout.ms should be higher
> than
> > > 30
> > > > > seconds given that we previously would reset the clock once the
> batch
> > > was
> > > > > sent. The value should be large enough that batches are not expired
> > due
> > > > to
> > > > > expected events like a new leader being elected due to broker
> > failure.
> > > > > Would it make sense to use a conservative value like 120 seconds?
> > > > >
> > > > > 2. The producer currently throws an exception for configuration
> > > > > combinations that don't make sense. We should probably do the same
> > here
> > > > for
> > > > > consistency (the KIP currently proposes a log warning).
> > > > >
> > > > > 3. We should mention that we will not cancel in flight requests
> until
> > > the
> > > > > request timeout even though we'll expire the batch early if needed.
> > > > >
> > > > > I think we should start the vote tomorrow so that we have a chance
> of
> > > > > hitting the KIP freeze for 1.0.0.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Wed, Sep 6, 2017 at 1:03 AM, Sumant Tambe 
> > > wrote:
> > > > >
> > > > > > I've updated the kip-91 writeup
> > > > > >  > > > > > 91+Provide+Intuitive+User+Timeouts+in+The+Producer>
> > > > > > to capture some of the discussion here. Please confirm if it's
> > > > 

[jira] [Created] (KAFKA-5858) consumer.poll() shouldn't throw exception due to deserialization error

[Jun Rao (JIRA)]

Jun Rao created KAFKA-5858:
--

 Summary: consumer.poll() shouldn't throw exception due to 
deserialization error
 Key: KAFKA-5858
 URL: https://issues.apache.org/jira/browse/KAFKA-5858
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.11.0.0
Reporter: Jun Rao


Currently, the new consumer will throw an exception in poll() if it hits a 
deserialization error. The consumer then can't make progress from this point 
on. It will be better to throw the deserialization exception only when the 
key/value of the ConsumerRecord is accessed, like the old consumer does.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5820) Remove unneeded synchronized keyword in StreamThread

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-5820.
--
   Resolution: Fixed
Fix Version/s: 1.0.0

Issue resolved by pull request 3777
[https://github.com/apache/kafka/pull/3777]

> Remove unneeded synchronized keyword in StreamThread
> 
>
> Key: KAFKA-5820
> URL: https://issues.apache.org/jira/browse/KAFKA-5820
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ted Yu
>Priority: Minor
> Fix For: 1.0.0
>
>
> There are three methods in StreamThread which have unnecessary synchronized 
> keyword since the variable accessed, state, is volatile :
> isRunningAndNotRebalancing
> isRunning
> shutdown
> synchronized keyword can be dropped for these methods.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3777: KAFKA-5820 Remove unneeded synchronized keyword in...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3777


---

[DISCUSS] KIP-196: Add metrics to Kafka Connect framework

[Randall Hauch]

Hi everyone.

I've created a new KIP to add metrics to the Kafka Connect framework:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework

The KIP approval deadline is looming, so if you're interested in Kafka
Connect metrics please review and provide feedback as soon as possible. I'm
interested not only in whether the metrics are sufficient and appropriate,
but also in whether the MBean naming conventions are okay.

Best regards,

Randall

Jenkins build is back to normal : kafka-trunk-jdk8 #1992

[Apache Jenkins Server]

See 

[GitHub] kafka pull request #3812: KAFKA-5657 Connect REST API should include the con...

[tedyu]

GitHub user tedyu opened a pull request:

https://github.com/apache/kafka/pull/3812

KAFKA-5657 Connect REST API should include the connector type when 
describing a connector

Embed the type of connector in ConnectorInfo

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tedyu/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3812.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3812


commit d51a65c81c576eb6a542adf92e9ab79d51f304d5
Author: tedyu 
Date:   2017-09-07T23:04:48Z

KAFKA-5657 Connect REST API should include the connector type when 
describing a connector




---

Re: [VOTE] KIP-192 : Provide cleaner semantics when idempotence is enabled

[Apurva Mehta]

Thanks for the votes!

Jason: I updated the KIP so that the messageFormatVersion field is int8.

Guozhang: The type of the config field is a purely internal concept. The
public API of the KafkaProducer takes a Map type or a
Properties type (which is a map of String to String). So this change in the
type doesn't change anything from an external point of view.

Thanks,
Apurva

On Thu, Sep 7, 2017 at 2:45 PM, Guozhang Wang  wrote:

> +1.
>
> A quick clarification question regarding the compatibility plan as for "The
> legacy values for `enable.idempotence` will be interpreted as follows by
> the new producer: true will mean required, false will mean off."
>
> Right now "enable.idempotence" is defined as Type.BOOLEAN while we are
> likely to change it to Type.STRING, is that considered compatible?
>
>
> Guozhang
>
>
> On Thu, Sep 7, 2017 at 2:25 PM, Jason Gustafson 
> wrote:
>
> > +1. Thanks for the KIP. One nit: we could use int8 to represent the
> message
> > format version. That is how it is represented in the messages themselves.
> >
> > -Jason
> >
> > On Thu, Sep 7, 2017 at 1:51 PM, Apurva Mehta 
> wrote:
> >
> > > Hi,
> > >
> > > I'd like to start a vote for KIP-192:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
> > >
> > > Thanks,
> > > Apurva
> > >
> >
>
>
>
> --
> -- Guozhang
>

[jira] [Created] (KAFKA-5857) Excessive heap usage on controller node during reassignment

[Raoufeh Hashemian (JIRA)]

Raoufeh Hashemian created KAFKA-5857:


 Summary: Excessive heap usage on controller node during 
reassignment
 Key: KAFKA-5857
 URL: https://issues.apache.org/jira/browse/KAFKA-5857
 Project: Kafka
  Issue Type: Bug
  Components: controller
Affects Versions: 0.11.0.0
 Environment: CentOs 7, Java 1.8
Reporter: Raoufeh Hashemian
 Attachments: CPU.png, disk_write_x.png, memory.png, 
reassignment_plan.txt

I was trying to expand our kafka cluster of 6 broker nodes to 12 broker nodes. 
Before expansion, we had a single topic with 960 partitions and a replication 
factor of 3. So each node had 480 partitions. The size of data in each node was 
3TB . 
To do the expansion, I submitted a partition reassignment plan (see attached 
file for the current/new assignments). The plan was optimized to minimize data 
movement and be rack aware. 

When I submitted the plan, it took approximately 3 hours to complete data 
movement after that, it started deleting source partitions (I say this based on 
the number of file descriptors) and rebalancing leaders which has never been 
successful. Meanwhile, the heap usage in the controller node started to peak up 
with a large slope (along with long GC times) and it took 5 hours for the 
controller to go out of memory and another controller started to have the same 
behaviour for another 4 hours. at this time the zookeeper ran out of disk and 
the service stopped.

To recover from this condition:
1) removed zk logs to free up disk and restarted all 3 zk nodes
2) deleted /kafka/admin/reassign_partitions node from zk
3) had to do unclean restarts of kafka service on oom controller nodes which 
took 3 hours to complete  . After this stage there was still 676 under 
replicated partitions.
4) Do a clean restart on all 12 broker nodes.

After step 5 , number of under replicated nodes went to 0.


So I was wondering if this memory footprint from controller is expected to 1k 
partitions ? Did we do sth wrong or it is a bug?

Attached are some resource usage graph during this 30 hours event and the 
reassignment plan.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3618: KAFKA-5698: Sort processor nodes based on its sub-...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3618


---

Re: [VOTE] KIP-192 : Provide cleaner semantics when idempotence is enabled

[Guozhang Wang]

+1.

A quick clarification question regarding the compatibility plan as for "The
legacy values for `enable.idempotence` will be interpreted as follows by
the new producer: true will mean required, false will mean off."

Right now "enable.idempotence" is defined as Type.BOOLEAN while we are
likely to change it to Type.STRING, is that considered compatible?


Guozhang


On Thu, Sep 7, 2017 at 2:25 PM, Jason Gustafson  wrote:

> +1. Thanks for the KIP. One nit: we could use int8 to represent the message
> format version. That is how it is represented in the messages themselves.
>
> -Jason
>
> On Thu, Sep 7, 2017 at 1:51 PM, Apurva Mehta  wrote:
>
> > Hi,
> >
> > I'd like to start a vote for KIP-192:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
> >
> > Thanks,
> > Apurva
> >
>



-- 
-- Guozhang

Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and introduction of TopicDeletePolicy

[Roger Hoover]

Edoardo, thanks for the KIP.  I think it's a good idea overall.

+1 especially for including Session/Principal in the API.  (#2 mentioned by
Ismael)

Also, the AlterPolicy should get the same info as create + delete (#4).

Cheers,

Roger

On Thu, Sep 7, 2017 at 8:43 AM, Ismael Juma  wrote:

> Hi Tom,
>
> Maybe we can discuss that as part of KIP-179. I don't think we should
> introduce a separate interface for the changes suggested here.
>
> Ismael
>
> On Tue, Sep 5, 2017 at 7:05 PM, Tom Bentley  wrote:
>
> > Hi Edoardo,
> >
> > KIP-179 will support changing topic configurations via the AdminClient
> and
> > this would open a loophole to avoid the current CreateTopicPolicy:
> >
> > 1. Create a topic that sticks to the policy
> > 2. Modify the topic after creation to something which would have violated
> > the policy.
> >
> > For this reason KIP-179 proposes to apply the create topic policy to
> topic
> > modification too. If there were going to be a new "V2" interface (as
> > opposed to changing the RequestMetadata) I think it would be beneficial
> if
> > the name reflected that it applied to updates too.
> > CreateOrAlterTopicPolicy? Though obviously that name doesn't reflect that
> > its heritage in the original CreateTopicPolicy.
> >
> > Cheers,
> >
> > Tom
> >
> > On 5 September 2017 at 18:48, Edoardo Comar  wrote:
> >
> > > Hi Ismael,
> > > I was on leave for a long while. I will update the KIP.
> > >
> > > Edo
> > >
> > > On 5 September 2017 at 11:42, Ismael Juma  wrote:
> > >
> > > > Hi Edoardo,
> > > >
> > > > Do you intend to update the KIP to avoid the introduction of another
> > > > interface?
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Jun 23, 2017 at 5:47 PM, Edoardo Comar 
> > > wrote:
> > > >
> > > > > Thanks for the thoughts Ismael
> > > > >
> > > > > > 1. Have you considered extending RequestMetadata with the
> > additional
> > > > > > information you need? We could add Cluster to it, which has topic
> > > > > > assignment information, for example. This way, there would be no
> > need
> > > > > for a
> > > > > > V2 interface.
> > > > >
> > > > > 1. I hadn't thought of it - but this seems a feasible alternative.
> > > > >
> > > > > The XXXPolicy.RequestMetadata could be enriched to include more
> > > > > information -
> > > > > for backward compatibility with existing Policies we would only add
> > > > > methods to these classes.
> > > > >
> > > > > Presumably the number of Admin Request is not huge so we should not
> > be
> > > > too
> > > > > worried
> > > > > about the overhead of populating a org.apache.kafka.common.Cluster
> > for
> > > > > every admin request , right ?
> > > > >
> > > > > The only feature I can't see how to easily support with the
> enriched
> > > > > XXXPolicy.RequestMetadata is how to check
> > > > > for a topic being marked for deletion using the Cluster information
> > > > >
> > > > > This check is useful for us in excluding such outliers when
> counting
> > > the
> > > > > number of partitions.
> > > > >
> > > > >
> > > > > > 2. Something else that could be useful is passing an instance of
> > > > > `Session`
> > > > > > so that one can provide custom behaviour depending on the logged
> in
> > > > > user.
> > > > > > Would this be useful?
> > > > > 2. Definitely I would expect that a general solution includes the
> > > Session
> > > > > or the KafkaPrincipal associated with it
> > > > > (the latter may be a simpler dependency for the
> > > XXXPolicy.RequestMetadata
> > > > >
> > > > >
> > > > > > 3. For the delete case, we may consider passing a class instead
> of
> > > just
> > > > > a
> > > > > > string to the validate method so that we have options if we need
> to
> > > > > extend
> > > > > > it.
> > > > >
> > > > > 3. Agree, we should have the DeletePolicy define its
> RequestMetadata
> > > > > class, too
> > > > >
> > > > >
> > > > > > 4. Do we want to enhance the AlterConfigs policy as well?
> > > > >
> > > > > 4. I don't see why not :-)
> > > > >
> > > > >
> > > > >
> > > > > thanks
> > > > > Edo
> > > > > --
> > > > >
> > > > > Edoardo Comar
> > > > >
> > > > > IBM Message Hub
> > > > >
> > > > >
> > > > > isma...@gmail.com wrote on 22/06/2017 15:05:06:
> > > > >
> > > > > > From: Ismael Juma 
> > > > > > To: dev@kafka.apache.org
> > > > > > Date: 22/06/2017 15:05
> > > > > > Subject: Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and
> > > > > > introduction of TopicDeletePolicy
> > > > > > Sent by: isma...@gmail.com
> > > > > >
> > > > > > Thanks for the KIP, Edoardo. A few comments:
> > > > > >
> > > > >
> > > > > >
> > > > >
> > > > > >
> > > > >
> > > > > >
> > > > >
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Thu, Jun 22, 2017 at 2:41 PM, Edoardo Comar <
> eco...@uk.ibm.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We've drafted 

[jira] [Resolved] (KAFKA-5698) Sort processor node based on name suffix in TopologyDescription.toString()

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-5698.
--
   Resolution: Fixed
Fix Version/s: 1.0.0

Issue resolved by pull request 3618
[https://github.com/apache/kafka/pull/3618]

> Sort processor node based on name suffix in TopologyDescription.toString()
> --
>
> Key: KAFKA-5698
> URL: https://issues.apache.org/jira/browse/KAFKA-5698
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>  Labels: newbie++
> Fix For: 1.0.0
>
>
> Today when we print the topology via the {{Topology#describe()#toString}}, 
> the processor nodes are not sorted inside the 
> {{TopologyDescription.toString()}} function. For example, for the word count 
> demo topology we get:
> {code}
> Sub-topologies: 
>   Sub-topology: 0
> Processor: KSTREAM-FILTER-05(stores: []) --> 
> KSTREAM-SINK-04 <-- KSTREAM-MAP-02
> Source: KSTREAM-SOURCE-00(topics: streams-wordcount-input) --> 
> KSTREAM-FLATMAPVALUES-01
> Processor: KSTREAM-FLATMAPVALUES-01(stores: []) --> 
> KSTREAM-MAP-02 <-- KSTREAM-SOURCE-00
> Processor: KSTREAM-MAP-02(stores: []) --> 
> KSTREAM-FILTER-05 <-- KSTREAM-FLATMAPVALUES-01
> Sink: KSTREAM-SINK-04(topic: Counts-repartition) <-- 
> KSTREAM-FILTER-05
>   Sub-topology: 1
> Source: KSTREAM-SOURCE-06(topics: Counts-repartition) --> 
> KSTREAM-AGGREGATE-03
> Sink: KSTREAM-SINK-08(topic: streams-wordcount-output) <-- 
> KTABLE-TOSTREAM-07
> Processor: KTABLE-TOSTREAM-07(stores: []) --> 
> KSTREAM-SINK-08 <-- KSTREAM-AGGREGATE-03
> Processor: KSTREAM-AGGREGATE-03(stores: [Counts]) --> 
> KTABLE-TOSTREAM-07 <-- KSTREAM-SOURCE-06
> {code}
> While ideally we want:
> {code}
> Sub-topologies: 
>   Sub-topology: 0
> Source: KSTREAM-SOURCE-00(topics: streams-wordcount-input) --> 
> KSTREAM-FLATMAPVALUES-01
> Processor: KSTREAM-FLATMAPVALUES-01(stores: []) --> 
> KSTREAM-MAP-02 <-- KSTREAM-SOURCE-00
> Processor: KSTREAM-MAP-02(stores: []) --> 
> KSTREAM-FILTER-05 <-- KSTREAM-FLATMAPVALUES-01
> Processor: KSTREAM-FILTER-05(stores: []) --> 
> KSTREAM-SINK-04 <-- KSTREAM-MAP-02
> Sink: KSTREAM-SINK-04(topic: Counts-repartition) <-- 
> KSTREAM-FILTER-05
>   Sub-topology: 1
> Source: KSTREAM-SOURCE-06(topics: Counts-repartition) --> 
> KSTREAM-AGGREGATE-03
> Processor: KSTREAM-AGGREGATE-03(stores: [Counts]) --> 
> KTABLE-TOSTREAM-07 <-- KSTREAM-SOURCE-06
> Processor: KTABLE-TOSTREAM-07(stores: []) --> 
> KSTREAM-SINK-08 <-- KSTREAM-AGGREGATE-03
> Sink: KSTREAM-SINK-08(topic: streams-wordcount-output) <-- 
> KTABLE-TOSTREAM-07
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

[Guozhang Wang]

Hi Tom,

The updated part in "AdminClient:electPreferredLeaders()" looks reasonable
to me. If there is no objections from the voted committer by end of the
day, I think you can mark it as accepted.


Guozhang


On Wed, Sep 6, 2017 at 7:42 AM, Tom Bentley  wrote:

> Unfortunately I've had to make a small change to the
> ElectPreferredLeadersResult, because exposing a Map KafkaFuture> was incompatible with the case where
> electPreferredLeaders() was called with a null partitions argument. The
> change exposes methods to access the map which return futures, rather than
> exposing the map (and crucially its keys) directly.
>
> This is described in more detail in the [DISCUSS] thread.
>
> Please take a look and recast your votes:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-183+-+Change+
> PreferredReplicaLeaderElectionCommand+to+use+AdminClient#KIP-183-
> ChangePreferredReplicaLeaderElectionCommandtouseAdminClient-AdminClient:
> electPreferredLeaders()
>
> Thanks,
>
> Tom
>
> On 4 September 2017 at 10:52, Ismael Juma  wrote:
>
> > Hi Tom,
> >
> > You can update the KIP for minor things like that. Worth updating the
> > thread if it's something that is done during the PR review.
> >
> > With regards to exceptions, yes, that's definitely desired. I filed a
> JIRA
> > a while back for this:
> >
> > https://issues.apache.org/jira/browse/KAFKA-5445
> >
> > Ideally, new methods that we add would have this so that we don't
> increase
> > the tech debt that already exists.
> >
> > Ismael
> >
> > On Mon, Sep 4, 2017 at 10:11 AM, Tom Bentley 
> > wrote:
> >
> > > Hi Jun,
> > >
> > > You're correct about those other expected errors. If it's OK to update
> > the
> > > KIP after the vote I'll add those.
> > >
> > > But this makes me wonder about the value of documenting expected errors
> > in
> > > the Javadocs for the AdminClient (on the Results class, to be
> specific).
> > > Currently we don't do this, but it would be helpful for people using
> the
> > > AdminClient to know the kinds of errors they should expect, for testing
> > > purposes for example. On the other hand it's a maintenance burden.
> Should
> > > we start documenting likely errors like this?
> > >
> > > Cheers,
> > >
> > > Tom
> > >
> > > On 4 September 2017 at 10:10, Tom Bentley 
> wrote:
> > >
> > > > I see three +1s, no +0s and no -1, so the vote passes.
> > > >
> > > > Thanks to those who voted and/or commented on the discussion thread.
> > > >
> > > > On 1 September 2017 at 07:36, Gwen Shapira 
> wrote:
> > > >
> > > >> Thank you! +1 (binding).
> > > >>
> > > >> On Thu, Aug 31, 2017 at 9:48 AM Jun Rao  wrote:
> > > >>
> > > >> > Hi, Tom,
> > > >> >
> > > >> > Thanks for the KIP. +1. Just one more minor comment. It seems that
> > the
> > > >> > ElectPreferredLeadersResponse
> > > >> > should expect at least 3 other types of errors : (1) request
> timeout
> > > >> > exception, (2) leader rebalance in-progress exception, (3) can't
> > move
> > > to
> > > >> > the preferred replica exception (i.e., preferred replica not in
> sync
> > > >> yet).
> > > >> >
> > > >> > Jun
> > > >> >
> > > >> > On Tue, Aug 29, 2017 at 8:56 AM, Tom Bentley <
> t.j.bent...@gmail.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Hi all,
> > > >> > >
> > > >> > > I would like to start the vote on KIP-183 which will provide an
> > > >> > AdminClient
> > > >> > > interface for electing the preferred replica, and refactor the
> > > >> > > kafka-preferred-replica-election.sh tool to use this interface.
> > > More
> > > >> > > details here:
> > > >> > >
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 183+-+Change+
> > > >> > > PreferredReplicaLeaderElectionCommand+to+use+AdminClient
> > > >> > >
> > > >> > >
> > > >> > > Regards,
> > > >> > >
> > > >> > > Tom
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>



-- 
-- Guozhang

Re: 1.0.0 KIPs Update

[Guozhang Wang]

Actually my bad, there is already a voting thread and you asked people to
recast a vote on a small change.

On Thu, Sep 7, 2017 at 2:27 PM, Guozhang Wang  wrote:

> Hi Tom,
>
> It seems KIP-183 is still in the discussion phase, and voting has not been
> started?
>
>
> Guozhang
>
>
> On Thu, Sep 7, 2017 at 1:13 AM, Tom Bentley  wrote:
>
>> Would it be possible to add KIP-183 to the list too, please?
>>
>> Thanks,
>>
>> Tom
>>
>> On 6 September 2017 at 22:04, Guozhang Wang  wrote:
>>
>> > Hi Vahid,
>> >
>> > Yes I have just added it while sending this email :)
>> >
>> >
>> > Guozhang
>> >
>> > On Wed, Sep 6, 2017 at 1:54 PM, Vahid S Hashemian <
>> > vahidhashem...@us.ibm.com
>> > > wrote:
>> >
>> > > Hi Guozhang,
>> > >
>> > > Thanks for the heads-up.
>> > >
>> > > Can KIP-163 be added to the list?
>> > > The proposal for this KIP is accepted, and the PR is ready for review.
>> > >
>> > > Thanks.
>> > > --Vahid
>> > >
>> > >
>> > >
>> > > From:   Guozhang Wang 
>> > > To: "dev@kafka.apache.org" 
>> > > Date:   09/06/2017 01:45 PM
>> > > Subject:1.0.0 KIPs Update
>> > >
>> > >
>> > >
>> > > Hello folks,
>> > >
>> > > This is a heads up on 1.0.0 progress:
>> > >
>> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.a
>> > > pache.org_confluence_pages_viewpage.action-3FpageId-3D717649
>> > > 13=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_
>> > > xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=bLvgeykOujjty9joOuWXD4wZab
>> > > o1CV0pULY4eqBxqzk=90UN7ejzCQmdPOyRR_2z304xLUSBCtOYi0KqhAo4EyU=
>> > >
>> > >
>> > > We have one week left towards the KIP deadline, which is Sept. 13th.
>> > There
>> > > are still a lot of KIPs that under discussion / voting process. For
>> the
>> > > KIP
>> > > proposer, please keep in mind that the voting has to be done before
>> the
>> > > deadline in order to be added into the coming release.
>> > >
>> > >
>> > > Thanks,
>> > > -- Guozhang
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> > --
>> > -- Guozhang
>> >
>>
>
>
>
> --
> -- Guozhang
>



-- 
-- Guozhang

Re: 1.0.0 KIPs Update

[Guozhang Wang]

Hi Tom,

It seems KIP-183 is still in the discussion phase, and voting has not been
started?


Guozhang


On Thu, Sep 7, 2017 at 1:13 AM, Tom Bentley  wrote:

> Would it be possible to add KIP-183 to the list too, please?
>
> Thanks,
>
> Tom
>
> On 6 September 2017 at 22:04, Guozhang Wang  wrote:
>
> > Hi Vahid,
> >
> > Yes I have just added it while sending this email :)
> >
> >
> > Guozhang
> >
> > On Wed, Sep 6, 2017 at 1:54 PM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com
> > > wrote:
> >
> > > Hi Guozhang,
> > >
> > > Thanks for the heads-up.
> > >
> > > Can KIP-163 be added to the list?
> > > The proposal for this KIP is accepted, and the PR is ready for review.
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > > From:   Guozhang Wang 
> > > To: "dev@kafka.apache.org" 
> > > Date:   09/06/2017 01:45 PM
> > > Subject:1.0.0 KIPs Update
> > >
> > >
> > >
> > > Hello folks,
> > >
> > > This is a heads up on 1.0.0 progress:
> > >
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.a
> > > pache.org_confluence_pages_viewpage.action-3FpageId-3D717649
> > > 13=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_
> > > xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=bLvgeykOujjty9joOuWXD4wZab
> > > o1CV0pULY4eqBxqzk=90UN7ejzCQmdPOyRR_2z304xLUSBCtOYi0KqhAo4EyU=
> > >
> > >
> > > We have one week left towards the KIP deadline, which is Sept. 13th.
> > There
> > > are still a lot of KIPs that under discussion / voting process. For the
> > > KIP
> > > proposer, please keep in mind that the voting has to be done before the
> > > deadline in order to be added into the coming release.
> > >
> > >
> > > Thanks,
> > > -- Guozhang
> > >
> > >
> > >
> > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
-- Guozhang

Re: [VOTE] KIP-192 : Provide cleaner semantics when idempotence is enabled

[Jason Gustafson]

+1. Thanks for the KIP. One nit: we could use int8 to represent the message
format version. That is how it is represented in the messages themselves.

-Jason

On Thu, Sep 7, 2017 at 1:51 PM, Apurva Mehta  wrote:

> Hi,
>
> I'd like to start a vote for KIP-192:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
>
> Thanks,
> Apurva
>

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

[Jun Rao]

Hi, Tom,

It seems that it's useful to know whether the leader is balanced for each
partition in ElectPreferredLeadersResult, instead of just being attempted?

Thanks,

Jun

On Wed, Sep 6, 2017 at 4:03 PM, Colin McCabe  wrote:

> On Wed, Sep 6, 2017, at 01:18, Tom Bentley wrote:
> > Hi Colin,
> >
> > Thanks for taking the time to respond.
> >
> > On 5 September 2017 at 22:22, Colin McCabe  wrote:
> >
> > > ...
> > > Why does there need to be a map at all in the API?
> >
> >
> > From a purely technical PoV there doesn't, but doing something else would
> > make the API inconsistent with other similar AdminClient *Results
> > classes,
> > which all expose a Map directly.
> >
> >
> > > Why not just have
> > > something like this:
> > >
> >
> > I agree this would be a better solution. I will update the KIP and ask
> > people to vote again. (Is that the right process?)
> >
> > It might be worth bearing this in mind for future AdminClient APIs:
> > Exposing a Map directly means you can't retrofit handling a null argument
> > to mean "all the things", whereas wrapping the map would allow that.
>
> That's a good point.
>
> I guess the important thing to keep in mind is that if you return a map
> from a results class, it has to be instantiated eagerly.  It has to be
> something you know before any RPCs are made, async actions are
> performed, etc.
>
> best,
> Colin
>
> >
> > Thanks again,
> >
> > Tom
>

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

[Gwen Shapira]

+1 (binding)

Looking forward to see how connector implementations use this in practice :)

On Thu, Sep 7, 2017 at 3:49 PM Randall Hauch  wrote:

> I'd like to open the vote for KIP-131:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector
>
> Thanks to Florian for submitting the KIP and the implementation, and to
> everyone else that helped review.
>
> Best regards,
>
> Randall
>

[jira] [Resolved] (KAFKA-3033) Reassigning partition stuck in progress

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3033.
--
Resolution: Duplicate

 Resolving this as a duplicate of KAFKA-4914. 

> Reassigning partition stuck in progress
> ---
>
> Key: KAFKA-3033
> URL: https://issues.apache.org/jira/browse/KAFKA-3033
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.9.0.0
> Environment: centos 7.2
>Reporter: Leo Xuzhang Lin
>Assignee: Neha Narkhede
>Priority: Critical
>  Labels: reliability
>
> We were trying to increase the replication factor on a test topic we've 
> created. 
> We followed the documentation's instruction:
> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor
> and received:
> ```
> Current partition replica assignment
> {"version":1,"partitions":[{"topic":"test","partition":0,"replicas":[1]}]}
> Save this to use as the --reassignment-json-file option during rollback
> Successfully started reassignment of partitions 
> {"version":1,"partitions":[{"topic":"test"
> ,"partition":0,"replicas":["1","2","3"]}]}
> ```
> After that whenever we try verify, it is stuck on:
> ```
> Status of partition reassignment:
> Reassignment of partition [test,0] is still in progress
> ```
> - We tried restarting the cluster and it still did not work.
> - The topic has 1 partition
> - The zookeeper /admin/reassign_partitions znode is empty



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

[Apurva Mehta]

I agree with what Ismael said. Having both retries and delivery.timeout.ms
is confusing, and thus the goal is to not have a retries option at all once
idempotence is fully battle tested and has become the entrenched default.

Until that time, it makes sense to expire batch earlier than
delivery.timeout.ms if retries have been exhausted.

Thanks,
Apurva


On Thu, Sep 7, 2017 at 6:07 AM, Ismael Juma  wrote:

> Good question regarding retries Sumant. A few comments:
>
> 1. Defaulting to MAX_INT makes sense in the context of delivery.timeout.ms
> ,
> but introduces the possibility of reordering with the default max.in.flight
> of 5. Personally, I think reordering is better than dropping the message
> altogether (if we keep retries=0), but it's worth noting this.
>
> 2. I agree that we should expire on whichever of retries and
> delivery.timeout.ms is exhausted first for 1.0.0.
>
> 3. Once KIP-185 lands (post 1.0.0), we should consider deprecating and
> eventually removing the retries config to simplify things (it won't have
> much use then).
>
> 4. With regards to the case where the broker replies quickly with an error,
> we need to understand a bit more what the error is. For any kind of
> connection issue, we now have exponential backoff. For the case where an
> error code is returned, it depends on whether the error is retriable or
> not. For the former, it probably makes sense to keep retrying as it's
> supposed to be a transient issue. If we think it would make sense to apply
> exponential backoff, we could also consider that. So, I'm not sure retries
> has much use apart from compatibility and the retries=0 case (for now).
>
> Ismael
>
> On Wed, Sep 6, 2017 at 11:14 PM, Jun Rao  wrote:
>
> > Hi, Sumant,
> >
> > The diagram in the wiki seems to imply that delivery.timeout.ms doesn't
> > include the batching time.
> >
> > For retries, probably we can just default it to MAX_INT?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, Sep 6, 2017 at 10:26 AM, Sumant Tambe  wrote:
> >
> > > 120 seconds default sounds good to me. Throwing ConfigException instead
> > of
> > > WARN is fine. Added clarification that the producer waits the full
> > > request.timeout.ms for the in-flight request. This implies that user
> > might
> > > be notified of batch expiry while a batch is still in-flight.
> > >
> > > I don't recall if we discussed our point of view that existing configs
> > like
> > > retries become redundant/deprecated with this feature. IMO, retries
> > config
> > > becomes meaningless due to the possibility of incorrect configs like
> > > delivery.timeout.ms > linger.ms + retries * (request..timeout.ms +
> > > retry.backoff.ms), retries should be basically interpreted as MAX_INT?
> > > What
> > > will be the default?
> > >
> > > So do we ignore retries config or throw a ConfigException if weirdness
> > like
> > > above is detected?
> > >
> > > -Sumant
> > >
> > >
> > > On 5 September 2017 at 17:34, Ismael Juma  wrote:
> > >
> > > > Thanks for updating the KIP, Sumant. A couple of points:
> > > >
> > > > 1. I think the default for delivery.timeout.ms should be higher than
> > 30
> > > > seconds given that we previously would reset the clock once the batch
> > was
> > > > sent. The value should be large enough that batches are not expired
> due
> > > to
> > > > expected events like a new leader being elected due to broker
> failure.
> > > > Would it make sense to use a conservative value like 120 seconds?
> > > >
> > > > 2. The producer currently throws an exception for configuration
> > > > combinations that don't make sense. We should probably do the same
> here
> > > for
> > > > consistency (the KIP currently proposes a log warning).
> > > >
> > > > 3. We should mention that we will not cancel in flight requests until
> > the
> > > > request timeout even though we'll expire the batch early if needed.
> > > >
> > > > I think we should start the vote tomorrow so that we have a chance of
> > > > hitting the KIP freeze for 1.0.0.
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Sep 6, 2017 at 1:03 AM, Sumant Tambe 
> > wrote:
> > > >
> > > > > I've updated the kip-91 writeup
> > > > >  > > > > 91+Provide+Intuitive+User+Timeouts+in+The+Producer>
> > > > > to capture some of the discussion here. Please confirm if it's
> > > > sufficiently
> > > > > accurate. Feel free to edit it if you think some explanation can be
> > > > better
> > > > > and has been agreed upon here.
> > > > >
> > > > > How do you proceed from here?
> > > > >
> > > > > -Sumant
> > > > >
> > > > > On 30 August 2017 at 12:59, Jun Rao  wrote:
> > > > >
> > > > > > Hi, Jiangjie,
> > > > > >
> > > > > > I mis-understood Jason's approach earlier. It does seem to be a
> > good
> > > > one.
> > > > > > We still need to calculate the selector timeout based on the
> > > remaining
> > > > > > 

[jira] [Resolved] (KAFKA-4138) Producer data write network traffic to kafka brokers is increased for 50%

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4138.
--
Resolution: Cannot Reproduce

Closing inactive issue. Similar issues are fixed in newer versions. Pl reopen 
if you think the issue still exists


> Producer data write network traffic to kafka brokers is increased for 50%
> -
>
> Key: KAFKA-4138
> URL: https://issues.apache.org/jira/browse/KAFKA-4138
> Project: Kafka
>  Issue Type: Improvement
>  Components: producer 
>Affects Versions: 0.9.0.1
> Environment: Redhat Enterprise 6.5
>Reporter: Zane Zhang
>  Labels: performance
>
> After we upgraded kafka from 0.8.2 to 0.9.0.1, it is observed that network 
> output traffic from producers to kafka is increased for 50% with same message 
> payload and same data pressure to producers. 
> We are using snappy compression, StringSerializer, other producer parameters 
> are listed as below:
> request.required.acks=1
> retries=3
> retry.backoff.ms=300
> batch.size=2*1024*1024
> max.request.size=20971520
> request.timeout.ms=2000



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[VOTE] KIP-192 : Provide cleaner semantics when idempotence is enabled

[Apurva Mehta]

Hi,

I'd like to start a vote for KIP-192:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled

Thanks,
Apurva

[jira] [Resolved] (KAFKA-4190) kafka-reassign-partitions does not report syntax problem in json

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4190.
--
Resolution: Duplicate

Resolving this duplicated KAFKA-4914.  KAFKA-4914 PR is being reviewed now. 

> kafka-reassign-partitions does not report syntax problem in json
> 
>
> Key: KAFKA-4190
> URL: https://issues.apache.org/jira/browse/KAFKA-4190
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Nikolay Sokolov
>Priority: Trivial
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> When specifying invalid json file, kafka-reassign-partitions fails with error 
> "file is empty" instead of reporting syntax error.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-4119) Get topic offset with Kafka SSL

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4119.
--
Resolution: Cannot Reproduce

 Please reopen if the issue still exists. 


> Get topic offset with Kafka SSL
> ---
>
> Key: KAFKA-4119
> URL: https://issues.apache.org/jira/browse/KAFKA-4119
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.1
> Environment: Linux
>Reporter: zhang shuai
>  Labels: kafka-acl, offset, ssl
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> I have a kafka cluster with SSL when i trun off ACL,i can get offset form 
> kafka-consumer-offset-checker.sh
> [root@node128 kafka_2.11-0.10.0.1]# bin/kafka-consumer-offset-checker.sh 
> --zookeeper localhost:2181/kafka-test --group consumer-group-1 --topic 
> testtopic
> [2016-09-03 14:03:51,722] WARN WARNING: ConsumerOffsetChecker is deprecated 
> and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand 
> instead. (kafka.tools.ConsumerOffsetChecker$)
> Group   Topic  Pid Offset  logSize
>  Lag Owner
> consumer-group-1 testtopic  0   99981   99981 
>   0   none
> Configuration like this:
> ssl.keystore.location=/opt/ssl_key/server.keystore.jks
> ssl.keystore.password=xdata123
> ssl.key.password=xdata123
> ssl.truststore.location=/opt/ssl_key/server.truststore.jks
> ssl.truststore.password=xdata123
> ssl.client.auth=required
> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
> ssl.keystore.type=JKS
> ssl.truststore.type=JKS
> security.inter.broker.protocol=SSL
> But when i want to use ACL and add configuration
> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> super.users=User:CN=node128,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown;User:CN=node129,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown
> Then I use kafka-acls.sh to add principal for topic testtopic and group 
> consumer-group-1，
> I cannot get result of kafka offset from kafka-consumer-offset-checker.sh. Is 
> there something deferent in ACL? How can I get topics` offset in kafka ACL 
> model?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-4020) Kafka consumer stop taking messages from kafka server

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4020.
--
Resolution: Fixed

 Please reopen if the issue still exists. 


> Kafka consumer stop taking messages from kafka server
> -
>
> Key: KAFKA-4020
> URL: https://issues.apache.org/jira/browse/KAFKA-4020
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.0
>Reporter: Shawn He
>
> It feels like the similar issue of KAFKA-2978, even though I haven't verified 
> if it is caused by the same events. How do I check on that? 
> I have a client that works fine using kafka 0.8.2.1, and can run months 
> without any issue. However, after I upgraded to use kafka 0.10.0.0, it's very 
> repeatable that the client will work for the first 4 hours, and then stopped 
> working. The producer side has no issue, as the data still comes in to the 
> kafka server. 
> I was using Java library kafka.consumer.Consumer.createJavaConsumerConnector 
> and kafka.consumer.KafkaStream class for the access to the kafka server.
> Any help is appreciated.
> Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-1993) Enable topic deletion as default

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1993.
--
Resolution: Fixed

Resolving this as duplicate of KAFKA-5384

> Enable topic deletion as default
> 
>
> Key: KAFKA-1993
> URL: https://issues.apache.org/jira/browse/KAFKA-1993
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
> Attachments: KAFKA-1993.patch
>
>
> Since topic deletion is now throughly tested and works as well as most Kafka 
> features, we should enable it by default.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-3113) Kafka simple consumer inconsistent result

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3113.
--
Resolution: Cannot Reproduce

may be seed broker is different for different topics. simpleconsumer should 
point to leader broker of the topic. Please reopen if the issue still exists. 


> Kafka simple consumer inconsistent result
> -
>
> Key: KAFKA-3113
> URL: https://issues.apache.org/jira/browse/KAFKA-3113
> Project: Kafka
>  Issue Type: Bug
>Reporter: Goutam Chowdhury
>
> I am trying to read kafka messages using spart api in batch mode.To achieve  
> this , I need start and last offset of the mentioned topic. To get start and 
> last offset I am creating simple consumer with below code
> -
> var consumer = new SimpleConsumer(seedBroker, seedBrokerPort, 10, 64 * 
> 1024, clientName);
> var topicAndPartition = new TopicAndPartition(topic, partition.toInt)
> var requestInfo = new HashMap[TopicAndPartition,PartitionOffsetRequestInfo];
>  requestInfo.put(topicAndPartition, new PartitionOffsetRequestInfo(whichTime, 
> 1))
>  logger.info("requestInfo - " + requestInfo);
> var request = new kafka.javaapi.OffsetRequest(requestInfo, 
> OffsetRequest.CurrentVersion, clientName);
> var response =consumer.getOffsetsBefore(request);
> ---
> I am using BIG IP addresses for seed broker  which actually has four brokers 
> . Now if I do spark submit then some times it get exception with below 
> response 
> response - OffsetResponse(0,Map([ecp.demo.patents.control,0] -> error: 
> kafka.common.UnknownTopicOrPartitionException offsets: ))
> but some time i get proper response. could anybody explain why it is like 
> that? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-2022) simpleconsumer.fetch(req) throws a java.nio.channels.ClosedChannelException: null exception when the original leader fails instead of being trapped in the fetchResponse

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2022.
--
Resolution: Won't Fix

I think we need to catch the exception and retry with a new leader.  Pl reopen 
if you think the issue still exists


> simpleconsumer.fetch(req) throws a java.nio.channels.ClosedChannelException: 
> null exception when the original leader fails instead of being trapped in the 
> fetchResponse api while consuming messages
> -
>
> Key: KAFKA-2022
> URL: https://issues.apache.org/jira/browse/KAFKA-2022
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.2.1
> Environment: 3 linux nodes with both zookeepr & brokers running under 
> respective users on each..
>Reporter: Muqeet Mohammed Ali
>Assignee: Neha Narkhede
>
> simpleconsumer.fetch(req) throws a java.nio.channels.ClosedChannelException: 
> null exception when the original leader fails, instead of being trapped in 
> the fetchResponse api while consuming messages. My understanding was that any 
> fetch failures can be found via fetchResponse.hasError() call and then be 
> handled to fetch new leader in this case. Below is the relevant code snippet 
> from the simple consumer with comments marking the line causing 
> exception..can you please comment on this?
> if (simpleconsumer == null) {
>   simpleconsumer = new 
> SimpleConsumer(leaderAddress.getHostName(), leaderAddress.getPort(), 
> consumerTimeout,
>   consumerBufferSize, 
> consumerId);
> }
> FetchRequest req = new FetchRequestBuilder().clientId(getConsumerId())
>   .addFetch(topic, partition, 
> offsetManager.getTempOffset(), consumerBufferSize)
>   // Note: the fetchSize might need to be increased
>   // if large batches are written to Kafka
>   .build();
> // exception is throw at the below line
> FetchResponse fetchResponse = simpleconsumer.fetch(req);
> if (fetchResponse.hasError()) {
>   numErrors++;
> etc...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-trunk-jdk8 #1991

[Apache Jenkins Server]

See 


Changes:

[me] KAFKA-4585: Lower the Minimum Required ACL Permission of OffsetFetch

--
[...truncated 3.85 MB...]
org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringInt 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testValidators STARTED

org.apache.kafka.common.config.ConfigDefTest > testValidators PASSED

org.apache.kafka.common.config.ConfigDefTest > testValidateCannotParse STARTED

org.apache.kafka.common.config.ConfigDefTest > testValidateCannotParse PASSED

org.apache.kafka.common.config.ConfigDefTest > testValidate STARTED

org.apache.kafka.common.config.ConfigDefTest > testValidate PASSED

org.apache.kafka.common.config.ConfigDefTest > 
testInternalConfigDoesntShowUpInDocs STARTED

org.apache.kafka.common.config.ConfigDefTest > 
testInternalConfigDoesntShowUpInDocs PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultString STARTED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultString PASSED

org.apache.kafka.common.config.ConfigDefTest > testSslPasswords STARTED

org.apache.kafka.common.config.ConfigDefTest > testSslPasswords PASSED

org.apache.kafka.common.config.ConfigDefTest > testBaseConfigDefDependents 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testBaseConfigDefDependents 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringClass 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringClass 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringShort 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringShort 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefault STARTED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefault PASSED

org.apache.kafka.common.config.ConfigDefTest > toRst STARTED

org.apache.kafka.common.config.ConfigDefTest > toRst PASSED

org.apache.kafka.common.config.ConfigDefTest > testCanAddInternalConfig STARTED

org.apache.kafka.common.config.ConfigDefTest > testCanAddInternalConfig PASSED

org.apache.kafka.common.config.ConfigDefTest > testMissingRequired STARTED

org.apache.kafka.common.config.ConfigDefTest > testMissingRequired PASSED

org.apache.kafka.common.config.ConfigDefTest > 
testParsingEmptyDefaultValueForStringFieldShouldSucceed STARTED

org.apache.kafka.common.config.ConfigDefTest > 
testParsingEmptyDefaultValueForStringFieldShouldSucceed PASSED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringPassword 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringPassword 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringList 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringList 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringLong 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringLong 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringBoolean 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringBoolean 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testNullDefaultWithValidator 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testNullDefaultWithValidator 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testMissingDependentConfigs 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testMissingDependentConfigs 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringDouble 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringDouble 
PASSED

org.apache.kafka.common.config.ConfigDefTest > toEnrichedRst STARTED

org.apache.kafka.common.config.ConfigDefTest > toEnrichedRst PASSED

org.apache.kafka.common.config.ConfigDefTest > testDefinedTwice STARTED

org.apache.kafka.common.config.ConfigDefTest > testDefinedTwice PASSED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringString 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testConvertValueToStringString 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testNestedClass STARTED

org.apache.kafka.common.config.ConfigDefTest > testNestedClass PASSED

org.apache.kafka.common.config.ConfigDefTest > testBadInputs STARTED

org.apache.kafka.common.config.ConfigDefTest > testBadInputs PASSED

org.apache.kafka.common.config.ConfigDefTest > testValidateMissingConfigKey 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testValidateMissingConfigKey 
PASSED

org.apache.kafka.common.config.AbstractConfigTest > testOriginalsWithPrefix 
STARTED

org.apache.kafka.common.config.AbstractConfigTest > testOriginalsWithPrefix 
PASSED

org.apache.kafka.common.config.AbstractConfigTest > testConfiguredInstances 
STARTED

[VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

[Randall Hauch]

I'd like to open the vote for KIP-131:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector

Thanks to Florian for submitting the KIP and the implementation, and to
everyone else that helped review.

Best regards,

Randall

Jenkins build is back to normal : kafka-trunk-jdk7 #2725

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-2595) Processor thread dies due to an uncaught NoSuchElementException

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2595.
--
Resolution: Fixed

See the discussion in KAFKA-1804

> Processor thread dies due to an uncaught NoSuchElementException
> ---
>
> Key: KAFKA-2595
> URL: https://issues.apache.org/jira/browse/KAFKA-2595
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Shaun Senecal
> Attachments: server.log.2015-09-23-09.gz
>
>
> We are getting uncaught exceptions which seem to kill the processor threads.  
> The end result is that we end up with a bunch of connections in CLOSE_WAIT 
> and eventually the broker is unable to respond or hits the max open files 
> ulimit.
> {noformat}
> [2015-09-23 09:54:33,687] ERROR Uncaught exception in thread 
> 'kafka-network-thread-9092-2': (kafka.utils.Utils$)
> java.util.NoSuchElementException: None.get
> at scala.None$.get(Option.scala:347)
> at scala.None$.get(Option.scala:345)
> at kafka.network.ConnectionQuotas.dec(SocketServer.scala:524)
> at kafka.network.AbstractServerThread.close(SocketServer.scala:165)
> at kafka.network.AbstractServerThread.close(SocketServer.scala:157)
> at kafka.network.Processor.close(SocketServer.scala:374)
> at kafka.network.Processor.processNewResponses(SocketServer.scala:406)
> at kafka.network.Processor.run(SocketServer.scala:318)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The issue appears to be the same as KAFKA-1577, except that its not happening 
> during shutdown.  We haven't been able to isolate when this happens, so we 
> dont have a good way to reproduce the issue.
> It also looks like KAFKA-2353 would work around the issue if it could be 
> back-ported, but the root cause should probably be fixed as well.
> - java version: 1.7.0_65
> - kafka version: 0.8.2.0
> - topics: 366
> - partitions: ~550 (a few 20 partition topics, and a bunch of 1 partition 
> topics)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

[Viktor Somogyi]

Hi Paolo,

To me it makes sense, let's do it :).

Viktor

On Thu, Sep 7, 2017 at 4:37 PM, Paolo Patierno  wrote:

> So as commented on the KAFKA-5723 JIRA my plan could be :
>
>
>   *   Using the KAFKA-3268 as umbrella for the others JIRAs related to
> tools refactoring
>   *   Regarding common components needed by developers involved on
> different tools, I will create a subtask in this JIRA (i.e. related to the
> new CommandOptions class)
>   *   For such subtask I'll open a PR with the work I have already done so
> a simple PR
>   *   The committers and the other "tools" developers will review the PR
> for making changes
>   *   When the PR will be merged the developers can continue on the
> different JIRAs related to different tools making their PRs
>
>
> Does this plan make sense ?
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>
>
> 
> From: isma...@gmail.com  on behalf of Ismael Juma <
> ism...@juma.me.uk>
> Sent: Thursday, September 7, 2017 10:56 AM
> To: dev@kafka.apache.org
> Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> AdminClient
>
> No, I'm suggesting that we think how can merge smaller PRs to trunk. Having
> a separate branch doesn't help as it can diverge from trunk and a committer
> would be needed to merge rebases, etc.
>
> Ismael
>
> On Thu, Sep 7, 2017 at 11:25 AM, Paolo Patierno 
> wrote:
>
> > Hi Ismael,
> >
> > first of all thanks for your reply.
> >
> > So as far as I understood having a branch in the Kafka repo could be
> > better for you as committer to validate small PRs from us and not a big
> one
> > at the end, right ?
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperience
> >
> >
> > 
> > From: isma...@gmail.com  on behalf of Ismael Juma <
> > ism...@juma.me.uk>
> > Sent: Thursday, September 7, 2017 10:19 AM
> > To: dev@kafka.apache.org
> > Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> > AdminClient
> >
> > I would also add that it would be easier to review if there were smaller
> > PRs than one big PR. So, it may be worth thinking how progress could be
> > made more incrementally.
> >
> > Ismael
> >
> > On Thu, Sep 7, 2017 at 11:17 AM, Tom Bentley 
> > wrote:
> >
> > > I can't speak for the committers, but there's nothing to stop you
> > > submitting PRs against each others branches. It just needs you to agree
> > > which of you will host the integration branch. This would be pretty
> much
> > > exactly the same developer experience as of the branch was in the main
> > > Kafak repo AFAICS, except the committers wouldn't have to be involved
> > with
> > > merging your PRs into your integration branch (which is probably a
> > benefit
> > > to both you and them).
> > >
> > > On 7 September 2017 at 10:57, Paolo Patierno 
> wrote:
> > >
> > > > Hi committers,
> > > >
> > > >
> > > > as already asked by Viktor on the JIRA yesterday can you give us a
> > > > feedback/advice on how to move on that ? Thanks !
> > > >
> > > >
> > > > Paolo Patierno
> > > > Senior Software Engineer (IoT) @ Red Hat
> > > > Microsoft MVP on Windows Embedded & IoT
> > > > Microsoft Azure Advisor
> > > >
> > > > Twitter : @ppatierno
> > > > Linkedin : paolopatierno
> > > > Blog : DevExperience
> > > >
> > > >
> > > > 
> > > > From: Paolo Patierno 
> > > > Sent: Thursday, August 17, 2017 3:18 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the
> > new
> > > > AdminClient
> > > >
> > > > As I started on working to develop the TopicCommand tool in Java
> (with
> > > > this PR), Andrey is
> working
> > > on
> > > > another one and he started to use some of my classes (mostly related
> to
> > > > handling command line arguments) just with copy/paste/modify so
> > > duplicating
> > > > the code.
> > > >
> > > > As mentioned in the JIRA discussion > > > apache.org/jira/browse/KAFKA-5723>, having a branch in the original
> > repo
> > > > could help us to have personal forks with such branch, opening PRs
> > > against
> > > > this branch and having the common 

[jira] [Resolved] (KAFKA-2623) Kakfa broker not deleting logs after configured retention time properly

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2623.
--
Resolution: Fixed

 Time-based log retention is enforced in KIP-33.  Pl reopen if you think the 
issue still exists


> Kakfa broker not deleting logs after configured retention time properly
> ---
>
> Key: KAFKA-2623
> URL: https://issues.apache.org/jira/browse/KAFKA-2623
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.2.0
> Environment: DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=12.04
> DISTRIB_CODENAME=precise
> DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"
> NAME="Ubuntu"
> VERSION="12.04.5 LTS, Precise Pangolin"
> ID=ubuntu
> ID_LIKE=debian
> PRETTY_NAME="Ubuntu precise (12.04.5 LTS)"
> VERSION_ID="12.04"
>Reporter: Hao Zhang
>Assignee: Jay Kreps
>
> Context:
> To get an accurate estimate on how much retention we have for each 
> topic/partition, we have a cron job iterating each topic/partition folder on 
> each node of a cluster, measuring the timestamp difference between the newest 
> and oldest log files. 
> Problem:
> We notice that it's very common that between leaders and followers, the time 
> differences are vastly different. On the leader the timestamp differences are 
> normally about a week (our retention policy), but on the follower the 
> timestamp differences can sometimes range between just a few hours to 2-3 
> days.
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 001536840178.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 001537497855.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 001538155208.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 001538811692.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 001539468154.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 001540122891.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 001540775681.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 001541430669.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 001542088333.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 001542746722.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 001543405006.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 001544062197.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 001544718413.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 001545374173.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 001546029145.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 001546686144.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 001547344190.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 001548001698.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 001548657672.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 001549312958.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 001549969014.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 001550623380.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 001551279821.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 001551937920.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 001552597354.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 001553256336.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 001553914505.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 001554571426.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 001555228277.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 001555882081.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 001556538902.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 001557196332.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 001557852974.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 001558510709.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 001559166839.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 001559823667.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 001560478631.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 001561136505.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 00156179.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 001562450149.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 001563107321.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 001563763826.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:52 001564420526.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:52 001565076456.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:52 001565735877.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:52 001566394151.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:52 001567051743.log
> -rw-rw-r-- 1 kloak kloak 256M Oct  4 05:52 001567709678.log
> 

[GitHub] kafka pull request #3811: KAFKA-5839: Upgrade Guide doc changes for KIP-130

[fhussonnois]

GitHub user fhussonnois opened a pull request:

https://github.com/apache/kafka/pull/3811

KAFKA-5839: Upgrade Guide doc changes for KIP-130



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fhussonnois/kafka KAFKA-5839

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3811.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3811


commit 08e5d5e76c852c3a52def1480daec96c4439d310
Author: Florian Hussonnois 
Date:   2017-09-07T19:19:04Z

KAFKA-5839: Upgrade Guide doc changes for KIP-130




---

[jira] [Resolved] (KAFKA-2166) Recreation breaks topic-list

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2166.
--
Resolution: Fixed

Deletion related issues are fixed in newer versions.  Pl reopen if you think 
the issue still exists


> Recreation breaks topic-list
> 
>
> Key: KAFKA-2166
> URL: https://issues.apache.org/jira/browse/KAFKA-2166
> Project: Kafka
>  Issue Type: Bug
>Reporter: Thomas Zimmer
>
> Hi here are the steps the reproduce the issue:
> * Create a topic called "test"
> * Delete the topic "test"
> * Recreate the topic "test" 
> What will happen is that you will see the topic in the topic-list but it's 
> marked as deleted:
>  ./kafka-topics.sh --list --zookeeper zookpeer1.dev, zookeeper2.dev
> test - marked for deletion
> Is there a way to fix it without having to delete everything? We also tried 
> several restarts



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3661: KAFKA-4585: Lower the Minimum Required ACL Permiss...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3661


---

[jira] [Resolved] (KAFKA-1563) High packet rate between brokers in kafka cluster.

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1563.
--
Resolution: Cannot Reproduce

 Pl reopen if you think the issue still exists


> High packet rate between brokers in kafka cluster.
> --
>
> Key: KAFKA-1563
> URL: https://issues.apache.org/jira/browse/KAFKA-1563
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Fedor Korotkiy
>
> On our kafka cluster with 3 brokers and input 40MB/s we see about 100K 
> packets/s traffic between brokers(not including consumers). Majority of 
> packets have small size(about 20bytes of data).
> I have found that kafka server sets TcpNoDelay option on all sockets. 
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/network/SocketServer.scala#L202
> And I think that causes the issue.
> Can you please explain current behavior and fix it/make it configurable?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-1455) Expose ConsumerOffsetChecker as an api instead of being command line only

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1455.
--
Resolution: Fixed

This is has been added to Java Admin API.

> Expose ConsumerOffsetChecker as an api instead of being command line only
> -
>
> Key: KAFKA-1455
> URL: https://issues.apache.org/jira/browse/KAFKA-1455
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Reporter: Arup Malakar
>Priority: Minor
>
> I find ConsumerOffsetChecker very useful when it comes to checking offset/lag 
> for a consumer group. It would be nice if it could be exposed as a class that 
> could be used from other programs instead of being only a command line too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-47) Create topic support and new ZK data structures for intra-cluster replication

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-47?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-47.

Resolution: Fixed

Closing this umbrella JIRA as all tasks are resolved.

> Create topic support and new ZK data structures for intra-cluster replication
> -
>
> Key: KAFKA-47
> URL: https://issues.apache.org/jira/browse/KAFKA-47
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jun Rao
>
> We need the DDL syntax for creating new topics. May need to use things like 
> javaCC. Also, we need to register new data structures in ZK accordingly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-641) ConsumerOffsetChecker breaks when using dns names x.b.com as opposed to raw public IP for broker.

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-641.
-
Resolution: Fixed

This is not relevant in newer versions.

> ConsumerOffsetChecker breaks when using dns names x.b.com as opposed to raw 
> public IP for broker.
> -
>
> Key: KAFKA-641
> URL: https://issues.apache.org/jira/browse/KAFKA-641
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.7.2
>Reporter: Juan Valencia
>Priority: Minor
>  Labels: offset
>
> private val BrokerIpPattern = """.*:(\d+\.\d+\.\d+\.\d+):(\d+$)""".r
>   // e.g., 127.0.0.1-1315436360737:127.0.0.1:9092
> if you have a domain name (such as ec2 instance: 
> ec2-111-11-111-111.compute-1.amazonaws.com), the script breaks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-252) Generalize getOffsetsBefore API to a new more general API getLeaderMetadata

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-252.
-
Resolution: Fixed

These API were handled in newer versions.

> Generalize getOffsetsBefore API to a new more general API getLeaderMetadata
> ---
>
> Key: KAFKA-252
> URL: https://issues.apache.org/jira/browse/KAFKA-252
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Neha Narkhede
>  Labels: project
>
> The relevant discussion is here - 
> https://issues.apache.org/jira/browse/KAFKA-238?focusedCommentId=13191350=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13191350
>  and on KAFKA-642
> We have an api that gets cluster-wide metadata (getTopicmetdata) which we use 
> for bootstraping knowledge about the cluster. But some things can only be 
> fetched from the leader for the partition.
> With replication, the metadata about log segments can only be returned by a 
> broker that hosts that partition locally. It will be good to expose log 
> segment metadata through a more general replica metadata API that in addition 
> to returning offsets, also returns other metadata like - number of log 
> segments, total size, last modified timestamp, highwater mark, and log end 
> offset.
> It would be good to do a wiki design on this and get consensus on that first 
> since this would be a public api.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-980) Crash during log recovery can cause full recovery to never run

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-980.
-
Resolution: Fixed

> Crash during log recovery can cause full recovery to never run
> --
>
> Key: KAFKA-980
> URL: https://issues.apache.org/jira/browse/KAFKA-980
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.7.1
>Reporter: Blake Smith
>
> After an unclean shutdown of the Kafka server, if the broker throws an 
> unhandled exception during log recovery, the broker can get in a state where 
> recovery never runs on a log file.
> We saw this problem manifest in production and is summarized on the mailing 
> list here: 
> http://mail-archives.apache.org/mod_mbox/kafka-users/201307.mbox/%3CCAKSpikjgp2sW2ycuf86JrjtAPxWBp92OOEmigVed=u=jfop...@mail.gmail.com%3E
> Because recovery state is not tracked explicitly, our kafka broker started 
> writing data even when the log files were not fully recovered. It feels to me 
> like a separate state flag for recovery should also be tracked in cases where 
> recovery does not fully run. What do you guys think?
> Steps to reproduce:
> 1. Shutdown the kafka broker
> 2. Create a directory named 'bogus' under the kafka log directory (won't 
> parse since it has no partition number)
> 3. Remove .kafka_cleanshutdown from the log directory to force a recovery
> 4. Start the kafka broker, observe:
> - Recovery will run on partition segments until it reaches the bogus 
> directory
> - Exception will be thrown during log loading from the bogus directory
> - Kafka will initiate a clean shutdown after the exception is thrown
> 5. Once the Kafka server is cleanly shutdown, start it again, observe:
> - Recovery will not try to run, since kafka was shutdown cleanly
> - Some partition log files have never been recovered
> 6. Remove the bogus log directory
> 7. Start Kafka broker, observe:
> - Recovery will not run
> - Kafka will start cleanly and begin accepting writes again, even though 
> recovery has never run and logs might be in a corrupt state



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-986) Topic Consumption Across multiple instances of consumer groups

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-986.
-
Resolution: Cannot Reproduce

> Topic Consumption Across multiple instances of consumer groups
> --
>
> Key: KAFKA-986
> URL: https://issues.apache.org/jira/browse/KAFKA-986
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.0
> Environment: Linux
>Reporter: Subbu Srinivasan
>Assignee: Neha Narkhede
>
> Folks,
> How can we simulate the notion of queues for consumers from multiple 
> instances?
> For eg: I have a topic log.
> From a single machine ( I tried from different machines also) I started two 
> consumers on same topic with same group id. Both the consumers get copes of 
> messages. 
> bin/kafka-console-consumer.sh --zookeeper kafka1:2181  --topic log --group 1
> bin/kafka-console-consumer.sh --zookeeper kafka1:2181  --topic log --group 1
> From the design section at http://kafka.apache.org/design.html
> 
> Each consumer process belongs to a consumer group and each message is 
> delivered to exactly one process within every consumer group. Hence a 
> consumer group allows many processes or machines to logically act as a single 
> consumer. The concept of consumer group is very powerful and can be used to 
> support the semantics of either a queue or topic as found in JMS. To support 
> queue semantics, we can put all consumers in a single consumer group, in 
> which case each message will go to a single consumer. 
> 
> Can someone elaborate on this?
> Thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-1093) Log.getOffsetsBefore(t, …) does not return the last confirmed offset before t

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1093.
--
Resolution: Invalid

relavent part of the code is not available now. So closing this now.

> Log.getOffsetsBefore(t, …) does not return the last confirmed offset before t
> -
>
> Key: KAFKA-1093
> URL: https://issues.apache.org/jira/browse/KAFKA-1093
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: Swapnil Ghike
>Assignee: Swapnil Ghike
> Attachments: KAFKA-1093.patch
>
>
> Let's say there are three log segments s1, s2, s3.
> In Log.getoffsetsBefore(t, …), the offsetTimeArray will look like - 
> [(s1.start, s1.lastModified), (s2.start, s2.lastModified), (s3.start, 
> s3.lastModified), (logEndOffset, currentTimeMs)].
> Let's say s2.lastModified < t < s3.lastModified. getOffsetsBefore(t, 1) will 
> return Seq(s2.start).
> However, we already know s3.firstAppendTime (s3.created in trunk). So, if 
> s3.firstAppendTime < t < s3.lastModified, we should rather return s3.start. 
> This also resolves another bug wherein the log has only one segment and 
> getOffsetsBefore() returns an empty Seq if the timestamp provided is less 
> than the lastModified of the only segment. We should rather return the 
> startOffset of the segment if the timestamp is greater than the 
> firstAppendTime of the segment.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Kafka Connect corrupts message schema while doing cast transformation

[Plotnikov Artem Igorevich]

Hi, there!

I have encountered the problem, that Kafka Connect's Cast transformation loses 
schema information (basically, schema name) while doing type casting. I have 
reproduced this problem with the following test in 
org.apache.kafka.connect.transforms.CastTest for current trunk repository 
branch:
```
@SuppressWarnings("unchecked")
@Test
public void castWholeRecordValueWithSchemaBooleanAndTimestampField() {
final Cast xform = new Cast.Value<>();
xform.configure(Collections.singletonMap(Cast.SPEC_CONFIG, 
"int64:boolean"));

SchemaBuilder builder = SchemaBuilder.struct();
builder.field("int64", Schema.INT64_SCHEMA);
builder.field("timestamp", Timestamp.SCHEMA);
Schema supportedTypesSchema = builder.build();

Struct recordValue = new Struct(supportedTypesSchema);
recordValue.put("int64", (long) 64);
recordValue.put("timestamp", new java.sql.Timestamp(0L));

SourceRecord transformed = xform.apply(new SourceRecord(null, null, 
"topic", 0,
supportedTypesSchema, recordValue));

assertNull(transformed.valueSchema());
assertEquals(true, ((Map) 
transformed.value()).get("int64"));
assertEquals(new java.sql.Timestamp(0L), ((Map) 
transformed.value()).get("timestamp"));
}
```
And this fails with the following exception:
```
org.apache.kafka.connect.errors.DataException: Invalid Java object for schema 
type INT64: class java.sql.Timestamp for field: "null"

at 
org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:240)
at 
org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:209)
at org.apache.kafka.connect.data.Struct.put(Struct.java:214)
at 
org.apache.kafka.connect.transforms.Cast.applyWithSchema(Cast.java:152)
at org.apache.kafka.connect.transforms.Cast.apply(Cast.java:108)
at 
org.apache.kafka.connect.transforms.CastTest.castWholeRecordValueWithSchemaBooleanAndTimestampField(CastTest.java:380)
```
This happens because Timestamp.SCHEMA has schema.type = 'INT64' and schema.name 
= "org.apache.kafka.connect.data.Timestamp", but 
org.apache.kafka.connect.transforms.Cast#getOrBuildSchema method copies only 
schema.type and rewrites schema.name with 'null'.

For example, such a behavior leads to connector failure while exporting data 
from a database having timestamp field and additional field we perform type 
casting on (I have a connector settings for PostgreSQL which leads to this 
problem).

Should I report a bug or there is something I misunderstand? I have patched a 
source code locally in order to fix this problem for my particular case and I 
am ready to prepare a more general path if it will be necessary.

I have also attached a git patch file with the failing test case.

Thanks,
Artem

[jira] [Resolved] (KAFKA-1197) Count of bytes or messages of a topic stored in kafka

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1197.
--
Resolution: Fixed

Size, LogStartOffset, LogEndOffset are exposed as metrics in newer verions.

> Count of bytes or messages of a topic stored in kafka
> -
>
> Key: KAFKA-1197
> URL: https://issues.apache.org/jira/browse/KAFKA-1197
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Affects Versions: 0.7.2, 0.8.0
>Reporter: Hanish Bansal
>Priority: Minor
>
> There should be direct way of measuring count of messages or bytes for a 
> topic stored in Kafka.
> There are already some very useful metrics like byteRate and messageRate 
> using what we can see count of bytes/messages coming into Kafka broker.
> I was looking for some jmx metrics that can give count of messages/bytes 
> stored in kafka.
> If we look into data stores like hbase we can see  how many messages are  
> stored in hbase or if we look into search engine like elasticsearch then also 
> we can see how many messages are stored/indexed in elasticsearch. In similar 
> way i was expecting that there should be some way to see count of  messages 
> or bytes for a topic stored in kafka without using any external tool.
> It will be really helpful if there is some support for this using some jmx 
> metric or by script.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3810: KAFKA-5816: [FOLLOW UP] - create ProducedInternal ...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3810

KAFKA-5816: [FOLLOW UP] - create ProducedInternal class

Create `ProducedInternal` and remove getters from `Produced`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-5816-follow-up

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3810.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3810


commit 3c27419193217d4b4e8bc7ec53cdcc07d552febe
Author: Damian Guy 
Date:   2017-09-07T17:26:28Z

create ProducedInternal class




---

[jira] [Created] (KAFKA-5856) AdminClient should be able to increase number of partitions

[Tom Bentley (JIRA)]

Tom Bentley created KAFKA-5856:
--

 Summary: AdminClient should be able to increase number of 
partitions
 Key: KAFKA-5856
 URL: https://issues.apache.org/jira/browse/KAFKA-5856
 Project: Kafka
  Issue Type: Improvement
Reporter: Tom Bentley
Assignee: Tom Bentley


It should be possible to increase the partition count using the AdminClient. 

See 
[KIP-195|https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%3A+AdminClient.increasePartitions]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

[Tom Bentley]

Hi Ismael,

OK, KIP-195 has been factored out.

Regarding the dynamic configs, I personally still think we should have a
> specific protocol API for that


Can you explain a little more why?

With regards to throttling, it would be
> worth thinking about a way where the throttling configs can be
> automatically removed without the user having to re-run the tool.
>

Isn't that just a matter of updating the topic configs for
(leader|follower).replication.throttled.replicas
at the same time we remove the reassignment znode? That leaves open the
question about whether to reset the rates at the same time.

But now I wonder what the broker configs
"(leader|follower).replication.throttled.rate"
are DynamicConfigs but the topic configs
"(leader|follower).replication.throttled.replicas"
are normal configs. Aren't the topic ones for the replicas just as dynamic
as the broker ones for the rate?

Thanks,

Tom


On 7 September 2017 at 17:24, Ismael Juma  wrote:

> Hi Tom,
>
> It won't be used within Kafka, but it's a public API that can be used by
> other projects. And the protocol can be used by non-Java clients. So, there
> is still value in including it.
>
> Regarding the dynamic configs, I personally still think we should have a
> specific protocol API for that. With regards to throttling, it would be
> worth thinking about a way where the throttling configs can be
> automatically removed without the user having to re-run the tool. So, yes,
> maybe it should be a separate KIP as well.
>
> Not sure if we need it in the template, but you're welcome to mention any
> dependencies when there are some.
>
> Thanks,
> Ismael
>
> On Thu, Sep 7, 2017 at 5:15 PM, Tom Bentley  wrote:
>
> > Hi Ismael,
> >
> > It would be good to get at least some of this into 1.0.0.
> >
> > We could put the increasePartitions() work into another KIP, but it would
> > be an unused AdminClient API in that release. The consumer of this API
> will
> > be the TopicsCommand when that get refactored to use the AdminClient.
> > That's something Paolo Patierno is proposing to do but afaik not in time
> > for 1.0.0. I don't think that's an issue, though, so I'll split out a
> > separate KIP.
> >
> > FWIW, we could also split out the proposal to support describeConfigs()
> and
> > alterConfigs() for dynamic configs into a separate KIP too. But that's
> also
> > a decision we can defer until we're looking at the remainder of KIP-179.
> >
> > Aside: I wonder if it would be a good idea for the KIP template to have a
> > "Depends on" field so people can more easily keep track of how multiple
> > in-flight KIPs depend on one another?
> >
> > Cheers,
> >
> > Tom
> >
> > On 7 September 2017 at 16:42, Ismael Juma  wrote:
> >
> > > Hi Tom,
> > >
> > > What do you think of moving `increasePartitionsCount` (or
> > > `increaseNumPartitions`) to a separate KIP? That is simple enough that
> we
> > > could potentially include it in 1.0.0. I'd be happy to review it.
> > > ReassignPartitions is more complex and we can probably aim to include
> > that
> > > in the January release. What do you think?
> > >
> > > Ismael
> > >
> > > On Wed, Sep 6, 2017 at 11:42 PM, Colin McCabe 
> > wrote:
> > >
> > > > On Wed, Sep 6, 2017, at 00:20, Tom Bentley wrote:
> > > > > Hi Ted and Colin,
> > > > >
> > > > > Thanks for the comments.
> > > > >
> > > > > It seems you're both happier with reassign rather than assign, so
> I'm
> > > > > happy
> > > > > to stick with that.
> > > > >
> > > > >
> > > > > On 5 September 2017 at 18:46, Colin McCabe 
> > wrote:
> > > > >
> > > > > > ...
> > > > >
> > > > >
> > > > > > Do we expect that reducing the number of partitions will ever be
> > > > > > supported by this API?  It seems like decreasing would require a
> > > > > > different API-- one which supported data movement, had a "check
> > > status
> > > > > > of this operation" feature, etc. etc.  If this API is only ever
> > going
> > > > to
> > > > > > be used to increase the number of partitions, I think we should
> > just
> > > > > > call it "increasePartitionCount" to avoid confusion.
> > > > > >
> > > > >
> > > > > I thought a little about the decrease possibility (hence the static
> > > > > factory
> > > > > methods on PartitionCount), but not really in enough detail. I
> > suppose
> > > a
> > > > > decrease process could look like this:
> > > > >
> > > > > 1. Producers start partitioning based on the decreased partition
> > count.
> > > > > 2. Consumers continue to consume from all partitions.
> > > > > 3. At some point all the records in the old partitions have expired
> > and
> > > > > they can be deleted.
> > > > >
> > > > > This wouldn't work for compacted topics. Of course a more
> aggressive
> > > > > strategy is also possible where we forcibly move data from the
> > > partitions
> > > > > to be deleted.
> > > > >
> > > > > Anyway, in either case the process would be long running, 

Re: [DISCUSS] KIP-195: AdminClient.increasePartitions

[Ted Yu]

Tom:
Looks good overall.

bq. for the topic from the AlterPartitionCountsResult

Please align the name of Result with current proposal.

Please also fill in JIRA number when you have it.

On Thu, Sep 7, 2017 at 9:38 AM, Tom Bentley  wrote:

> As suggested by Ismael, I've factored the increasePartitionCounts() API out
> of KIP-179 out into a separate KIP which hopefully can progress more
> quickly.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%3A+AdminClient.
> increasePartitions
>
> If you've looked at KIP-179 in the last few days there's no much new to see
> here, but if not you should find KIP-195 a lighter read.
>
> Cheers,
>
> Tom
>

[DISCUSS] KIP-195: AdminClient.increasePartitions

[Tom Bentley]

As suggested by Ismael, I've factored the increasePartitionCounts() API out
of KIP-179 out into a separate KIP which hopefully can progress more
quickly.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%3A+AdminClient.increasePartitions

If you've looked at KIP-179 in the last few days there's no much new to see
here, but if not you should find KIP-195 a lighter read.

Cheers,

Tom

[GitHub] kafka pull request #3809: KAFKA-5853: implement WindowedKStream

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3809

KAFKA-5853: implement WindowedKStream

Add the `WindowedKStream` interface and implementation of methods that 
don't require `Materialized`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kgrouped-stream-windowed-by

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3809.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3809


commit 78b99500f40eb31644f88e0effcaea6cfba0a3d0
Author: Damian Guy 
Date:   2017-09-07T15:54:38Z

implement WindowedKStream




---

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

[Ismael Juma]

Hi Tom,

It won't be used within Kafka, but it's a public API that can be used by
other projects. And the protocol can be used by non-Java clients. So, there
is still value in including it.

Regarding the dynamic configs, I personally still think we should have a
specific protocol API for that. With regards to throttling, it would be
worth thinking about a way where the throttling configs can be
automatically removed without the user having to re-run the tool. So, yes,
maybe it should be a separate KIP as well.

Not sure if we need it in the template, but you're welcome to mention any
dependencies when there are some.

Thanks,
Ismael

On Thu, Sep 7, 2017 at 5:15 PM, Tom Bentley  wrote:

> Hi Ismael,
>
> It would be good to get at least some of this into 1.0.0.
>
> We could put the increasePartitions() work into another KIP, but it would
> be an unused AdminClient API in that release. The consumer of this API will
> be the TopicsCommand when that get refactored to use the AdminClient.
> That's something Paolo Patierno is proposing to do but afaik not in time
> for 1.0.0. I don't think that's an issue, though, so I'll split out a
> separate KIP.
>
> FWIW, we could also split out the proposal to support describeConfigs() and
> alterConfigs() for dynamic configs into a separate KIP too. But that's also
> a decision we can defer until we're looking at the remainder of KIP-179.
>
> Aside: I wonder if it would be a good idea for the KIP template to have a
> "Depends on" field so people can more easily keep track of how multiple
> in-flight KIPs depend on one another?
>
> Cheers,
>
> Tom
>
> On 7 September 2017 at 16:42, Ismael Juma  wrote:
>
> > Hi Tom,
> >
> > What do you think of moving `increasePartitionsCount` (or
> > `increaseNumPartitions`) to a separate KIP? That is simple enough that we
> > could potentially include it in 1.0.0. I'd be happy to review it.
> > ReassignPartitions is more complex and we can probably aim to include
> that
> > in the January release. What do you think?
> >
> > Ismael
> >
> > On Wed, Sep 6, 2017 at 11:42 PM, Colin McCabe 
> wrote:
> >
> > > On Wed, Sep 6, 2017, at 00:20, Tom Bentley wrote:
> > > > Hi Ted and Colin,
> > > >
> > > > Thanks for the comments.
> > > >
> > > > It seems you're both happier with reassign rather than assign, so I'm
> > > > happy
> > > > to stick with that.
> > > >
> > > >
> > > > On 5 September 2017 at 18:46, Colin McCabe 
> wrote:
> > > >
> > > > > ...
> > > >
> > > >
> > > > > Do we expect that reducing the number of partitions will ever be
> > > > > supported by this API?  It seems like decreasing would require a
> > > > > different API-- one which supported data movement, had a "check
> > status
> > > > > of this operation" feature, etc. etc.  If this API is only ever
> going
> > > to
> > > > > be used to increase the number of partitions, I think we should
> just
> > > > > call it "increasePartitionCount" to avoid confusion.
> > > > >
> > > >
> > > > I thought a little about the decrease possibility (hence the static
> > > > factory
> > > > methods on PartitionCount), but not really in enough detail. I
> suppose
> > a
> > > > decrease process could look like this:
> > > >
> > > > 1. Producers start partitioning based on the decreased partition
> count.
> > > > 2. Consumers continue to consume from all partitions.
> > > > 3. At some point all the records in the old partitions have expired
> and
> > > > they can be deleted.
> > > >
> > > > This wouldn't work for compacted topics. Of course a more aggressive
> > > > strategy is also possible where we forcibly move data from the
> > partitions
> > > > to be deleted.
> > > >
> > > > Anyway, in either case the process would be long running, whereas the
> > > > increase case is fast, so the semantics are quite different. So I
> > agree,
> > > > lets rename the method to make clear that it's only for increasing
> the
> > > > partition count.
> > > >
> > > > >
> > > > > > Outstanding questions:
> > > > > >
> > > > > > 1. Is the proposed alterInterReplicaThrottle() API really better
> > than
> > > > > > changing the throttle via alterConfigs()?
> > > > >
> > > > > That's a good point.  I would argue that we should just use
> > > alterConfigs
> > > > > to set the broker configuration, rather than having a special RPC
> > just
> > > > > for this.
> > > > >
> > > >
> > > > Yes, I'm minded to agree.
> > > >
> > > > The reason I originally thought a special API might be better was if
> > > > people
> > > > felt that the DynamicConfig mechanism (which seems to exist only to
> > > > support
> > > > these throttles) was an implementation detail of the throttles. But I
> > now
> > > > realise that they're visible via kafka-topics.sh, so they're
> > effectively
> > > > already a public API.
> > > >
> > > >
> > > > >
> > > > > ...
> > > > > > Would it be a problem that
> > > > > > triggering the reassignment required ClusterAction on the

Re: [VOTE] KIP-189 - Improve principal builder interface and add support for SASL

[Jason Gustafson]

I am closing the vote. Here are the totals:

Binding: Ismael, Rajini, Jun, (Me)
Non-binding: Mayuresh, Manikumar, Mickael

Thanks all for the reviews!



On Wed, Sep 6, 2017 at 2:22 PM, Jason Gustafson  wrote:

> Hi All,
>
> When implementing this, I found that the SecurityProtocol class has some
> internal details which we might not want to expose to users (in particular
> to enable testing). Since it's still useful to know the security protocol
> in use in some cases, and since the security protocol names are already
> exposed in configuration (and hence cannot easily change), I have modified
> the method in AuthenticationContext to return the name of the security
> protocol instead. Let me know if there are any concerns with this change.
> Otherwise, I will close out the vote.
>
> Thanks,
> Jason
>
> On Tue, Sep 5, 2017 at 11:10 AM, Ismael Juma  wrote:
>
>> Thanks for the KIP, +1 (binding).
>>
>> Ismael
>>
>> On Wed, Aug 30, 2017 at 4:51 PM, Jason Gustafson 
>> wrote:
>>
>> > I'd like to open the vote for KIP-189:
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > 189%3A+Improve+principal+builder+interface+and+add+support+for+SASL.
>> > Thanks to everyone who helped review.
>> >
>> > -Jason
>> >
>>
>
>

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

[Tom Bentley]

Hi Ismael,

It would be good to get at least some of this into 1.0.0.

We could put the increasePartitions() work into another KIP, but it would
be an unused AdminClient API in that release. The consumer of this API will
be the TopicsCommand when that get refactored to use the AdminClient.
That's something Paolo Patierno is proposing to do but afaik not in time
for 1.0.0. I don't think that's an issue, though, so I'll split out a
separate KIP.

FWIW, we could also split out the proposal to support describeConfigs() and
alterConfigs() for dynamic configs into a separate KIP too. But that's also
a decision we can defer until we're looking at the remainder of KIP-179.

Aside: I wonder if it would be a good idea for the KIP template to have a
"Depends on" field so people can more easily keep track of how multiple
in-flight KIPs depend on one another?

Cheers,

Tom

On 7 September 2017 at 16:42, Ismael Juma  wrote:

> Hi Tom,
>
> What do you think of moving `increasePartitionsCount` (or
> `increaseNumPartitions`) to a separate KIP? That is simple enough that we
> could potentially include it in 1.0.0. I'd be happy to review it.
> ReassignPartitions is more complex and we can probably aim to include that
> in the January release. What do you think?
>
> Ismael
>
> On Wed, Sep 6, 2017 at 11:42 PM, Colin McCabe  wrote:
>
> > On Wed, Sep 6, 2017, at 00:20, Tom Bentley wrote:
> > > Hi Ted and Colin,
> > >
> > > Thanks for the comments.
> > >
> > > It seems you're both happier with reassign rather than assign, so I'm
> > > happy
> > > to stick with that.
> > >
> > >
> > > On 5 September 2017 at 18:46, Colin McCabe  wrote:
> > >
> > > > ...
> > >
> > >
> > > > Do we expect that reducing the number of partitions will ever be
> > > > supported by this API?  It seems like decreasing would require a
> > > > different API-- one which supported data movement, had a "check
> status
> > > > of this operation" feature, etc. etc.  If this API is only ever going
> > to
> > > > be used to increase the number of partitions, I think we should just
> > > > call it "increasePartitionCount" to avoid confusion.
> > > >
> > >
> > > I thought a little about the decrease possibility (hence the static
> > > factory
> > > methods on PartitionCount), but not really in enough detail. I suppose
> a
> > > decrease process could look like this:
> > >
> > > 1. Producers start partitioning based on the decreased partition count.
> > > 2. Consumers continue to consume from all partitions.
> > > 3. At some point all the records in the old partitions have expired and
> > > they can be deleted.
> > >
> > > This wouldn't work for compacted topics. Of course a more aggressive
> > > strategy is also possible where we forcibly move data from the
> partitions
> > > to be deleted.
> > >
> > > Anyway, in either case the process would be long running, whereas the
> > > increase case is fast, so the semantics are quite different. So I
> agree,
> > > lets rename the method to make clear that it's only for increasing the
> > > partition count.
> > >
> > > >
> > > > > Outstanding questions:
> > > > >
> > > > > 1. Is the proposed alterInterReplicaThrottle() API really better
> than
> > > > > changing the throttle via alterConfigs()?
> > > >
> > > > That's a good point.  I would argue that we should just use
> > alterConfigs
> > > > to set the broker configuration, rather than having a special RPC
> just
> > > > for this.
> > > >
> > >
> > > Yes, I'm minded to agree.
> > >
> > > The reason I originally thought a special API might be better was if
> > > people
> > > felt that the DynamicConfig mechanism (which seems to exist only to
> > > support
> > > these throttles) was an implementation detail of the throttles. But I
> now
> > > realise that they're visible via kafka-topics.sh, so they're
> effectively
> > > already a public API.
> > >
> > >
> > > >
> > > > ...
> > > > > Would it be a problem that
> > > > > triggering the reassignment required ClusterAction on the Cluster,
> > but
> > > > > throttling the assignment required Alter on the Topic? What if a
> > user had
> > > > > the former permission, but not the latter?
> > > >
> > > > We've been trying to reserve ClusterAction on Cluster for
> > > > broker-initiated operations.  Alter on Cluster is the ACL for "root
> > > > stuff" and I would argue that it should be what we use here.
> > > >
> > > > For reconfiguring the broker, I think we should follow KIP-133 and
> use
> > > > AlterConfigs on the Broker resource.  (Of course, if you just use the
> > > > existing alterConfigs call, you get this without any special effort.)
> > > >
> > >
> > > Yes, sorry, what I put in the email about authorisation wasn't what I
> put
> > > in the KIP (I revised the KIP after drafting the email and then forgot
> to
> > > update the email).
> > >
> > > Although KIP-133 proposes a Broker resource, I don't see one in the
> code
> > > and KIP-133 was supposedly delivered in 

Build failed in Jenkins: kafka-trunk-jdk8 #1990

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] KAFKA-5777; Add ducktape integration for Trogdor

--
[...truncated 4.89 MB...]
org.apache.kafka.streams.integration.GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableJoin PASSED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithIntermediateUserTopic STARTED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithIntermediateUserTopic PASSED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithoutIntermediateUserTopic STARTED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithoutIntermediateUserTopic PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.KStreamAggregationDedupIntegrationTest > 
shouldReduce STARTED

org.apache.kafka.streams.integration.KStreamAggregationDedupIntegrationTest > 
shouldReduce PASSED

org.apache.kafka.streams.integration.KStreamAggregationDedupIntegrationTest > 
shouldGroupByKey STARTED

Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and introduction of TopicDeletePolicy

[Ismael Juma]

Hi Tom,

Maybe we can discuss that as part of KIP-179. I don't think we should
introduce a separate interface for the changes suggested here.

Ismael

On Tue, Sep 5, 2017 at 7:05 PM, Tom Bentley  wrote:

> Hi Edoardo,
>
> KIP-179 will support changing topic configurations via the AdminClient and
> this would open a loophole to avoid the current CreateTopicPolicy:
>
> 1. Create a topic that sticks to the policy
> 2. Modify the topic after creation to something which would have violated
> the policy.
>
> For this reason KIP-179 proposes to apply the create topic policy to topic
> modification too. If there were going to be a new "V2" interface (as
> opposed to changing the RequestMetadata) I think it would be beneficial if
> the name reflected that it applied to updates too.
> CreateOrAlterTopicPolicy? Though obviously that name doesn't reflect that
> its heritage in the original CreateTopicPolicy.
>
> Cheers,
>
> Tom
>
> On 5 September 2017 at 18:48, Edoardo Comar  wrote:
>
> > Hi Ismael,
> > I was on leave for a long while. I will update the KIP.
> >
> > Edo
> >
> > On 5 September 2017 at 11:42, Ismael Juma  wrote:
> >
> > > Hi Edoardo,
> > >
> > > Do you intend to update the KIP to avoid the introduction of another
> > > interface?
> > >
> > > Ismael
> > >
> > > On Fri, Jun 23, 2017 at 5:47 PM, Edoardo Comar 
> > wrote:
> > >
> > > > Thanks for the thoughts Ismael
> > > >
> > > > > 1. Have you considered extending RequestMetadata with the
> additional
> > > > > information you need? We could add Cluster to it, which has topic
> > > > > assignment information, for example. This way, there would be no
> need
> > > > for a
> > > > > V2 interface.
> > > >
> > > > 1. I hadn't thought of it - but this seems a feasible alternative.
> > > >
> > > > The XXXPolicy.RequestMetadata could be enriched to include more
> > > > information -
> > > > for backward compatibility with existing Policies we would only add
> > > > methods to these classes.
> > > >
> > > > Presumably the number of Admin Request is not huge so we should not
> be
> > > too
> > > > worried
> > > > about the overhead of populating a org.apache.kafka.common.Cluster
> for
> > > > every admin request , right ?
> > > >
> > > > The only feature I can't see how to easily support with the enriched
> > > > XXXPolicy.RequestMetadata is how to check
> > > > for a topic being marked for deletion using the Cluster information
> > > >
> > > > This check is useful for us in excluding such outliers when counting
> > the
> > > > number of partitions.
> > > >
> > > >
> > > > > 2. Something else that could be useful is passing an instance of
> > > > `Session`
> > > > > so that one can provide custom behaviour depending on the logged in
> > > > user.
> > > > > Would this be useful?
> > > > 2. Definitely I would expect that a general solution includes the
> > Session
> > > > or the KafkaPrincipal associated with it
> > > > (the latter may be a simpler dependency for the
> > XXXPolicy.RequestMetadata
> > > >
> > > >
> > > > > 3. For the delete case, we may consider passing a class instead of
> > just
> > > > a
> > > > > string to the validate method so that we have options if we need to
> > > > extend
> > > > > it.
> > > >
> > > > 3. Agree, we should have the DeletePolicy define its RequestMetadata
> > > > class, too
> > > >
> > > >
> > > > > 4. Do we want to enhance the AlterConfigs policy as well?
> > > >
> > > > 4. I don't see why not :-)
> > > >
> > > >
> > > >
> > > > thanks
> > > > Edo
> > > > --
> > > >
> > > > Edoardo Comar
> > > >
> > > > IBM Message Hub
> > > >
> > > >
> > > > isma...@gmail.com wrote on 22/06/2017 15:05:06:
> > > >
> > > > > From: Ismael Juma 
> > > > > To: dev@kafka.apache.org
> > > > > Date: 22/06/2017 15:05
> > > > > Subject: Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and
> > > > > introduction of TopicDeletePolicy
> > > > > Sent by: isma...@gmail.com
> > > > >
> > > > > Thanks for the KIP, Edoardo. A few comments:
> > > > >
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > >
> > > >
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Thu, Jun 22, 2017 at 2:41 PM, Edoardo Comar 
> > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We've drafted "KIP-170: Enhanced TopicCreatePolicy and
> introduction
> > > of
> > > > > > TopicDeletePolicy" for discussion:
> > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 170%3A+Enhanced+
> > > > > > TopicCreatePolicy+and+introduction+of+TopicDeletePolicy
> > > > > >
> > > > > > Please take a look. Your feedback is welcome and much needed.
> > > > > >
> > > > > > Thanks,
> > > > > > Edoardo
> > > > > > --
> > > > > > Edoardo Comar
> > > > > > IBM Message Hub
> > > > > > eco...@uk.ibm.com
> > > > > > IBM UK Ltd, Hursley Park, SO21 2JN
> > > > 

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

[Ismael Juma]

Hi Tom,

What do you think of moving `increasePartitionsCount` (or
`increaseNumPartitions`) to a separate KIP? That is simple enough that we
could potentially include it in 1.0.0. I'd be happy to review it.
ReassignPartitions is more complex and we can probably aim to include that
in the January release. What do you think?

Ismael

On Wed, Sep 6, 2017 at 11:42 PM, Colin McCabe  wrote:

> On Wed, Sep 6, 2017, at 00:20, Tom Bentley wrote:
> > Hi Ted and Colin,
> >
> > Thanks for the comments.
> >
> > It seems you're both happier with reassign rather than assign, so I'm
> > happy
> > to stick with that.
> >
> >
> > On 5 September 2017 at 18:46, Colin McCabe  wrote:
> >
> > > ...
> >
> >
> > > Do we expect that reducing the number of partitions will ever be
> > > supported by this API?  It seems like decreasing would require a
> > > different API-- one which supported data movement, had a "check status
> > > of this operation" feature, etc. etc.  If this API is only ever going
> to
> > > be used to increase the number of partitions, I think we should just
> > > call it "increasePartitionCount" to avoid confusion.
> > >
> >
> > I thought a little about the decrease possibility (hence the static
> > factory
> > methods on PartitionCount), but not really in enough detail. I suppose a
> > decrease process could look like this:
> >
> > 1. Producers start partitioning based on the decreased partition count.
> > 2. Consumers continue to consume from all partitions.
> > 3. At some point all the records in the old partitions have expired and
> > they can be deleted.
> >
> > This wouldn't work for compacted topics. Of course a more aggressive
> > strategy is also possible where we forcibly move data from the partitions
> > to be deleted.
> >
> > Anyway, in either case the process would be long running, whereas the
> > increase case is fast, so the semantics are quite different. So I agree,
> > lets rename the method to make clear that it's only for increasing the
> > partition count.
> >
> > >
> > > > Outstanding questions:
> > > >
> > > > 1. Is the proposed alterInterReplicaThrottle() API really better than
> > > > changing the throttle via alterConfigs()?
> > >
> > > That's a good point.  I would argue that we should just use
> alterConfigs
> > > to set the broker configuration, rather than having a special RPC just
> > > for this.
> > >
> >
> > Yes, I'm minded to agree.
> >
> > The reason I originally thought a special API might be better was if
> > people
> > felt that the DynamicConfig mechanism (which seems to exist only to
> > support
> > these throttles) was an implementation detail of the throttles. But I now
> > realise that they're visible via kafka-topics.sh, so they're effectively
> > already a public API.
> >
> >
> > >
> > > ...
> > > > Would it be a problem that
> > > > triggering the reassignment required ClusterAction on the Cluster,
> but
> > > > throttling the assignment required Alter on the Topic? What if a
> user had
> > > > the former permission, but not the latter?
> > >
> > > We've been trying to reserve ClusterAction on Cluster for
> > > broker-initiated operations.  Alter on Cluster is the ACL for "root
> > > stuff" and I would argue that it should be what we use here.
> > >
> > > For reconfiguring the broker, I think we should follow KIP-133 and use
> > > AlterConfigs on the Broker resource.  (Of course, if you just use the
> > > existing alterConfigs call, you get this without any special effort.)
> > >
> >
> > Yes, sorry, what I put in the email about authorisation wasn't what I put
> > in the KIP (I revised the KIP after drafting the email and then forgot to
> > update the email).
> >
> > Although KIP-133 proposes a Broker resource, I don't see one in the code
> > and KIP-133 was supposedly delivered in 0.11. Can anyone fill in the
> > story
> > here? Is it simply because the functionality to update broker configs
> > hasn't been implemented yet?
>
> Look in
> ./clients/src/main/java/org/apache/kafka/common/config/
> ConfigResource.java,
> for the BROKER resource.  I bet you're looking at the Resource class
> used for ACLs, which is a different class.
>
> >
> > As currently proposed, both reassignPartitions() and
> > alterInterBrokerThrottle()
> > require Alter on the Cluster. If we used alterConfigs() to set the
> > throttles then we create a situation where the triggering of the
> > reassignment required Alter(Cluster), but the throttling required
> > Alter(Broker), and the user might have the former but not the latter. I
> > don't think this is likely to be a big deal in practice, but maybe others
> > disagree?
>
> Alter:Cluster is essentially root, though.  If you have Alter:Cluster
> and you don't have AlterConfigs:Broker, you can just create a new ACL
> giving it to yourself (creating and deleting ACLs is one of the powers
> of Alter:Cluster)
>
> cheers,
> Colin
>
> >
> >
> > > >
> > > > 2. Is reassignPartitions() really the best name? I find 

[jira] [Created] (KAFKA-5855) records-lag is always zero

[Mohsen Zainalpour (JIRA)]

Mohsen Zainalpour created KAFKA-5855:


 Summary: records-lag is always zero
 Key: KAFKA-5855
 URL: https://issues.apache.org/jira/browse/KAFKA-5855
 Project: Kafka
  Issue Type: Bug
  Components: consumer, metrics
Affects Versions: 0.10.1.0
Reporter: Mohsen Zainalpour
Priority: Minor


I use Prometheus JmxExporter to export Kafka consumer metrics but the value of 
records-lag is always zero.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5854) Handle SASL authentication failures as non-retriable exceptions in clients

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-5854:
-

 Summary: Handle SASL authentication failures as non-retriable 
exceptions in clients
 Key: KAFKA-5854
 URL: https://issues.apache.org/jira/browse/KAFKA-5854
 Project: Kafka
  Issue Type: Sub-task
  Components: clients
Reporter: Rajini Sivaram
Assignee: Vahid Hashemian
 Fix For: 1.0.0


Produce and consumer changes to avoid retries on authentication failures.

Details are in 
[KIP-152|https://cwiki.apache.org/confluence/display/KAFKA/KIP-152+-+Improve+diagnostics+for+SASL+authentication+failures]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3808: MINOR: Always specify the keystore type in system ...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/3808

MINOR: Always specify the keystore type in system tests

Also throw an exception if a null keystore type is seen
in `SecurityStore`. This should never happen.

The default keystore type has changed in Java 9 (
http://openjdk.java.net/jeps/229), so we need to
be explicit to have consistent behaviour across
Java versions.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka set-jks-explicitly-in-system-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3808.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3808


commit cc9d9b50736ce80e93d318485e2762976050204f
Author: Ismael Juma 
Date:   2017-09-07T14:37:58Z

MINOR: Always specify the keystore type in system tests

Also throw an exception if a null keystore type is seen
in `SecurityStore`. This should never happen.

The default keystore type has changed in Java 9 (
http://openjdk.java.net/jeps/229), so we need to
be explicit to have consistent behaviour across
Java versions.




---

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

[Paolo Patierno]

So as commented on the KAFKA-5723 JIRA my plan could be :


  *   Using the KAFKA-3268 as umbrella for the others JIRAs related to tools 
refactoring
  *   Regarding common components needed by developers involved on different 
tools, I will create a subtask in this JIRA (i.e. related to the new 
CommandOptions class)
  *   For such subtask I'll open a PR with the work I have already done so a 
simple PR
  *   The committers and the other "tools" developers will review the PR for 
making changes
  *   When the PR will be merged the developers can continue on the different 
JIRAs related to different tools making their PRs


Does this plan make sense ?


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience



From: isma...@gmail.com  on behalf of Ismael Juma 

Sent: Thursday, September 7, 2017 10:56 AM
To: dev@kafka.apache.org
Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new 
AdminClient

No, I'm suggesting that we think how can merge smaller PRs to trunk. Having
a separate branch doesn't help as it can diverge from trunk and a committer
would be needed to merge rebases, etc.

Ismael

On Thu, Sep 7, 2017 at 11:25 AM, Paolo Patierno  wrote:

> Hi Ismael,
>
> first of all thanks for your reply.
>
> So as far as I understood having a branch in the Kafka repo could be
> better for you as committer to validate small PRs from us and not a big one
> at the end, right ?
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>
>
> 
> From: isma...@gmail.com  on behalf of Ismael Juma <
> ism...@juma.me.uk>
> Sent: Thursday, September 7, 2017 10:19 AM
> To: dev@kafka.apache.org
> Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> AdminClient
>
> I would also add that it would be easier to review if there were smaller
> PRs than one big PR. So, it may be worth thinking how progress could be
> made more incrementally.
>
> Ismael
>
> On Thu, Sep 7, 2017 at 11:17 AM, Tom Bentley 
> wrote:
>
> > I can't speak for the committers, but there's nothing to stop you
> > submitting PRs against each others branches. It just needs you to agree
> > which of you will host the integration branch. This would be pretty much
> > exactly the same developer experience as of the branch was in the main
> > Kafak repo AFAICS, except the committers wouldn't have to be involved
> with
> > merging your PRs into your integration branch (which is probably a
> benefit
> > to both you and them).
> >
> > On 7 September 2017 at 10:57, Paolo Patierno  wrote:
> >
> > > Hi committers,
> > >
> > >
> > > as already asked by Viktor on the JIRA yesterday can you give us a
> > > feedback/advice on how to move on that ? Thanks !
> > >
> > >
> > > Paolo Patierno
> > > Senior Software Engineer (IoT) @ Red Hat
> > > Microsoft MVP on Windows Embedded & IoT
> > > Microsoft Azure Advisor
> > >
> > > Twitter : @ppatierno
> > > Linkedin : paolopatierno
> > > Blog : DevExperience
> > >
> > >
> > > 
> > > From: Paolo Patierno 
> > > Sent: Thursday, August 17, 2017 3:18 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the
> new
> > > AdminClient
> > >
> > > As I started on working to develop the TopicCommand tool in Java (with
> > > this PR), Andrey is working
> > on
> > > another one and he started to use some of my classes (mostly related to
> > > handling command line arguments) just with copy/paste/modify so
> > duplicating
> > > the code.
> > >
> > > As mentioned in the JIRA discussion > > apache.org/jira/browse/KAFKA-5723>, having a branch in the original
> repo
> > > could help us to have personal forks with such branch, opening PRs
> > against
> > > this branch and having the common code there (syncing the code with
> > forked
> > > repos).
> > >
> > > In conclusion the problem is : having more people working on different
> > > pieces of code which have some common code (still under development)
> not
> > > yet available in the original repo.
> > >
> > > At least it's the idea we had but maybe the committers have a different
> > > way to go in such situations. Any 

[jira] [Created] (KAFKA-5853) Add WindowedKStream interface and implemenation

[Damian Guy (JIRA)]

Damian Guy created KAFKA-5853:
-

 Summary: Add WindowedKStream interface and implemenation
 Key: KAFKA-5853
 URL: https://issues.apache.org/jira/browse/KAFKA-5853
 Project: Kafka
  Issue Type: Sub-task
Reporter: Damian Guy


Add the {{WindowedKStream}} interface and implementation. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : kafka-trunk-jdk8 #1989

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

[Ismael Juma]

Good question regarding retries Sumant. A few comments:

1. Defaulting to MAX_INT makes sense in the context of delivery.timeout.ms,
but introduces the possibility of reordering with the default max.in.flight
of 5. Personally, I think reordering is better than dropping the message
altogether (if we keep retries=0), but it's worth noting this.

2. I agree that we should expire on whichever of retries and
delivery.timeout.ms is exhausted first for 1.0.0.

3. Once KIP-185 lands (post 1.0.0), we should consider deprecating and
eventually removing the retries config to simplify things (it won't have
much use then).

4. With regards to the case where the broker replies quickly with an error,
we need to understand a bit more what the error is. For any kind of
connection issue, we now have exponential backoff. For the case where an
error code is returned, it depends on whether the error is retriable or
not. For the former, it probably makes sense to keep retrying as it's
supposed to be a transient issue. If we think it would make sense to apply
exponential backoff, we could also consider that. So, I'm not sure retries
has much use apart from compatibility and the retries=0 case (for now).

Ismael

On Wed, Sep 6, 2017 at 11:14 PM, Jun Rao  wrote:

> Hi, Sumant,
>
> The diagram in the wiki seems to imply that delivery.timeout.ms doesn't
> include the batching time.
>
> For retries, probably we can just default it to MAX_INT?
>
> Thanks,
>
> Jun
>
>
> On Wed, Sep 6, 2017 at 10:26 AM, Sumant Tambe  wrote:
>
> > 120 seconds default sounds good to me. Throwing ConfigException instead
> of
> > WARN is fine. Added clarification that the producer waits the full
> > request.timeout.ms for the in-flight request. This implies that user
> might
> > be notified of batch expiry while a batch is still in-flight.
> >
> > I don't recall if we discussed our point of view that existing configs
> like
> > retries become redundant/deprecated with this feature. IMO, retries
> config
> > becomes meaningless due to the possibility of incorrect configs like
> > delivery.timeout.ms > linger.ms + retries * (request..timeout.ms +
> > retry.backoff.ms), retries should be basically interpreted as MAX_INT?
> > What
> > will be the default?
> >
> > So do we ignore retries config or throw a ConfigException if weirdness
> like
> > above is detected?
> >
> > -Sumant
> >
> >
> > On 5 September 2017 at 17:34, Ismael Juma  wrote:
> >
> > > Thanks for updating the KIP, Sumant. A couple of points:
> > >
> > > 1. I think the default for delivery.timeout.ms should be higher than
> 30
> > > seconds given that we previously would reset the clock once the batch
> was
> > > sent. The value should be large enough that batches are not expired due
> > to
> > > expected events like a new leader being elected due to broker failure.
> > > Would it make sense to use a conservative value like 120 seconds?
> > >
> > > 2. The producer currently throws an exception for configuration
> > > combinations that don't make sense. We should probably do the same here
> > for
> > > consistency (the KIP currently proposes a log warning).
> > >
> > > 3. We should mention that we will not cancel in flight requests until
> the
> > > request timeout even though we'll expire the batch early if needed.
> > >
> > > I think we should start the vote tomorrow so that we have a chance of
> > > hitting the KIP freeze for 1.0.0.
> > >
> > > Ismael
> > >
> > > On Wed, Sep 6, 2017 at 1:03 AM, Sumant Tambe 
> wrote:
> > >
> > > > I've updated the kip-91 writeup
> > > >  > > > 91+Provide+Intuitive+User+Timeouts+in+The+Producer>
> > > > to capture some of the discussion here. Please confirm if it's
> > > sufficiently
> > > > accurate. Feel free to edit it if you think some explanation can be
> > > better
> > > > and has been agreed upon here.
> > > >
> > > > How do you proceed from here?
> > > >
> > > > -Sumant
> > > >
> > > > On 30 August 2017 at 12:59, Jun Rao  wrote:
> > > >
> > > > > Hi, Jiangjie,
> > > > >
> > > > > I mis-understood Jason's approach earlier. It does seem to be a
> good
> > > one.
> > > > > We still need to calculate the selector timeout based on the
> > remaining
> > > > > delivery.timeout.ms to call the callback on time, but we can
> always
> > > wait
> > > > > for an inflight request based on request.timeout.ms.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Tue, Aug 29, 2017 at 5:16 PM, Becket Qin 
> > > > wrote:
> > > > >
> > > > > > Yeah, I think expiring a batch but still wait for the response is
> > > > > probably
> > > > > > reasonable given the result is not guaranteed anyways.
> > > > > >
> > > > > > @Jun,
> > > > > >
> > > > > > I think the frequent PID reset may still be possible if we do not
> > > wait
> > > > > for
> > > > > > the in-flight response to return. 

Build failed in Jenkins: kafka-trunk-jdk7 #2724

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] KAFKA-5777; Add ducktape integration for Trogdor

--
[...truncated 921.93 KB...]

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica 
STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldSurviveFastLeaderChange STARTED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 

[GitHub] kafka pull request #3807: KAFKA-5852: Add filter, filterNot, mapValues and M...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3807

KAFKA-5852: Add filter, filterNot, mapValues and Materialized to KTable

Add overloads of `filter`, `filterNot`, `mapValues` that take 
`Materialized` as a param to `KTable`. Deprecate overloads using `storeName` 
and `storeSupplier`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka ktable-filter-map

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3807.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3807


commit 1ee7c5f22488b593748ebec4bf78bfa607349835
Author: Damian Guy 
Date:   2017-09-07T11:52:56Z

blah

commit 259127ff3daf1db77f1c2dcc57d50d643b02
Author: Damian Guy 
Date:   2017-09-07T12:34:43Z

KTable filter, filterNot, mapValues and Materialized




---

[jira] [Created] (KAFKA-5852) Add filter, filterNot, mapValues and Materialized to KTable

[Damian Guy (JIRA)]

Damian Guy created KAFKA-5852:
-

 Summary: Add filter, filterNot, mapValues and Materialized to 
KTable
 Key: KAFKA-5852
 URL: https://issues.apache.org/jira/browse/KAFKA-5852
 Project: Kafka
  Issue Type: Sub-task
Reporter: Damian Guy
Assignee: Damian Guy
 Fix For: 1.0.0


Add overloads of {{filter}}, {{filterNot}}, {{mapValues}} that take 
{{Materialized}} as a param to KTable. Deprecate overloads using {{storeName}} 
and {{storeSupplier}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3726: KAFKA-5777. Add ducktape integration for Trogdor

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3726


---

[jira] [Resolved] (KAFKA-5777) Add ducktape integration for the Trogdor Fault injection daemon

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-5777.
---
   Resolution: Fixed
Fix Version/s: 1.0.0

Issue resolved by pull request 3726
[https://github.com/apache/kafka/pull/3726]

> Add ducktape integration for the Trogdor Fault injection daemon
> ---
>
> Key: KAFKA-5777
> URL: https://issues.apache.org/jira/browse/KAFKA-5777
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Colin P. McCabe
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-trunk-jdk7 #2723

[Apache Jenkins Server]

See 


Changes:

[damian.guy] KAFKA-5844; add groupBy(selector, serialized) to Ktable

--
[...truncated 921.94 KB...]
kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica 
STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 

[GitHub] kafka pull request #3802: KAFKA-5844: add groupBy(selector, serialized) to K...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3802


---

[jira] [Resolved] (KAFKA-5844) Add groupBy(KeyValueMapper, Serialized) to KTable

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy resolved KAFKA-5844.
---
Resolution: Fixed

Issue resolved by pull request 3802
[https://github.com/apache/kafka/pull/3802]

> Add groupBy(KeyValueMapper, Serialized) to KTable
> -
>
> Key: KAFKA-5844
> URL: https://issues.apache.org/jira/browse/KAFKA-5844
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 1.0.0
>
>
> part of KIP-182
> add {{KTable#groupBy(KeyValueMapper, Serialized)}} and deprecate the overload 
> with {{Serde}} params



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

[Ismael Juma]

No, I'm suggesting that we think how can merge smaller PRs to trunk. Having
a separate branch doesn't help as it can diverge from trunk and a committer
would be needed to merge rebases, etc.

Ismael

On Thu, Sep 7, 2017 at 11:25 AM, Paolo Patierno  wrote:

> Hi Ismael,
>
> first of all thanks for your reply.
>
> So as far as I understood having a branch in the Kafka repo could be
> better for you as committer to validate small PRs from us and not a big one
> at the end, right ?
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>
>
> 
> From: isma...@gmail.com  on behalf of Ismael Juma <
> ism...@juma.me.uk>
> Sent: Thursday, September 7, 2017 10:19 AM
> To: dev@kafka.apache.org
> Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> AdminClient
>
> I would also add that it would be easier to review if there were smaller
> PRs than one big PR. So, it may be worth thinking how progress could be
> made more incrementally.
>
> Ismael
>
> On Thu, Sep 7, 2017 at 11:17 AM, Tom Bentley 
> wrote:
>
> > I can't speak for the committers, but there's nothing to stop you
> > submitting PRs against each others branches. It just needs you to agree
> > which of you will host the integration branch. This would be pretty much
> > exactly the same developer experience as of the branch was in the main
> > Kafak repo AFAICS, except the committers wouldn't have to be involved
> with
> > merging your PRs into your integration branch (which is probably a
> benefit
> > to both you and them).
> >
> > On 7 September 2017 at 10:57, Paolo Patierno  wrote:
> >
> > > Hi committers,
> > >
> > >
> > > as already asked by Viktor on the JIRA yesterday can you give us a
> > > feedback/advice on how to move on that ? Thanks !
> > >
> > >
> > > Paolo Patierno
> > > Senior Software Engineer (IoT) @ Red Hat
> > > Microsoft MVP on Windows Embedded & IoT
> > > Microsoft Azure Advisor
> > >
> > > Twitter : @ppatierno
> > > Linkedin : paolopatierno
> > > Blog : DevExperience
> > >
> > >
> > > 
> > > From: Paolo Patierno 
> > > Sent: Thursday, August 17, 2017 3:18 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the
> new
> > > AdminClient
> > >
> > > As I started on working to develop the TopicCommand tool in Java (with
> > > this PR), Andrey is working
> > on
> > > another one and he started to use some of my classes (mostly related to
> > > handling command line arguments) just with copy/paste/modify so
> > duplicating
> > > the code.
> > >
> > > As mentioned in the JIRA discussion > > apache.org/jira/browse/KAFKA-5723>, having a branch in the original
> repo
> > > could help us to have personal forks with such branch, opening PRs
> > against
> > > this branch and having the common code there (syncing the code with
> > forked
> > > repos).
> > >
> > > In conclusion the problem is : having more people working on different
> > > pieces of code which have some common code (still under development)
> not
> > > yet available in the original repo.
> > >
> > > At least it's the idea we had but maybe the committers have a different
> > > way to go in such situations. Any feedback is really appreciated :-)
> > >
> > >
> > > Thanks,
> > >
> > > Paolo
> > >
> > >
> > > Paolo Patierno
> > > Senior Software Engineer (IoT) @ Red Hat
> > > Microsoft MVP on Windows Embedded & IoT
> > > Microsoft Azure Advisor
> > >
> > > Twitter : @ppatierno
> > > Linkedin : paolopatierno
> > > Blog : DevExperience
> > >
> > >
> > > 
> > > From: Andrey Dyachkov 
> > > Sent: Thursday, August 17, 2017 3:10 PM
> > > To: dev@kafka.apache.org
> > > Subject: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> > > AdminClient
> > >
> > > Good day,
> > > I am quite new here, my apologies for being kind of pushy.
> > > We had discussion here https://issues.apache.org/
> jira/browse/KAFKA-5723,
> > > that having separate branch for implementing the whole change
> > > https://issues.apache.org/jira/browse/KAFKA-3268 would simplify dev
> > > process. Could some of committers give a hand here?
> > >
> > > Also could you review pr https://github.com/apache/kafka/pull/3671 and
> > add
> > > me to contributors list in order to add myself to ticket 

Build failed in Jenkins: kafka-trunk-jdk8 #1988

[Apache Jenkins Server]

See 


Changes:

[damian.guy] KAFKA-5650; add StateStoreBuilder interface and implementations

--
[...truncated 3.85 MB...]
org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryWithoutPasswordConfiguration PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > testClientMode STARTED

org.apache.kafka.common.security.ssl.SslFactoryTest > testClientMode PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryConfiguration STARTED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryConfiguration PASSED

org.apache.kafka.common.security.kerberos.KerberosNameTest > testParse STARTED

org.apache.kafka.common.security.kerberos.KerberosNameTest > testParse PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testPrincipalNameCanContainSeparator STARTED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testPrincipalNameCanContainSeparator PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testEqualsAndHashCode STARTED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testEqualsAndHashCode PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameOverride STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameOverride PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingOptionValue 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingOptionValue PASSED

org.apache.kafka.common.security.JaasContextTest > testSingleOption STARTED

org.apache.kafka.common.security.JaasContextTest > testSingleOption PASSED

org.apache.kafka.common.security.JaasContextTest > 
testNumericOptionWithoutQuotes STARTED

org.apache.kafka.common.security.JaasContextTest > 
testNumericOptionWithoutQuotes PASSED

org.apache.kafka.common.security.JaasContextTest > testConfigNoOptions STARTED

org.apache.kafka.common.security.JaasContextTest > testConfigNoOptions PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithWrongListenerName STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithWrongListenerName PASSED

org.apache.kafka.common.security.JaasContextTest > testNumericOptionWithQuotes 
STARTED

org.apache.kafka.common.security.JaasContextTest > testNumericOptionWithQuotes 
PASSED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionValue STARTED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionValue PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingLoginModule 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingLoginModule PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingSemicolon STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingSemicolon PASSED

org.apache.kafka.common.security.JaasContextTest > testMultipleOptions STARTED

org.apache.kafka.common.security.JaasContextTest > testMultipleOptions PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForClientWithListenerName STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForClientWithListenerName PASSED

org.apache.kafka.common.security.JaasContextTest > testMultipleLoginModules 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMultipleLoginModules 
PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingControlFlag 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingControlFlag PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameAndFallback STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameAndFallback PASSED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionName STARTED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionName PASSED

org.apache.kafka.common.security.JaasContextTest > testControlFlag STARTED

org.apache.kafka.common.security.JaasContextTest > testControlFlag PASSED

org.apache.kafka.common.security.plain.PlainSaslServerTest > 
noAuthorizationIdSpecified STARTED

org.apache.kafka.common.security.plain.PlainSaslServerTest > 
noAuthorizationIdSpecified PASSED

org.apache.kafka.common.security.plain.PlainSaslServerTest > 
authorizatonIdEqualsAuthenticationId STARTED

org.apache.kafka.common.security.plain.PlainSaslServerTest > 
authorizatonIdEqualsAuthenticationId PASSED

org.apache.kafka.common.security.plain.PlainSaslServerTest > 
authorizatonIdNotEqualsAuthenticationId STARTED

org.apache.kafka.common.security.plain.PlainSaslServerTest > 
authorizatonIdNotEqualsAuthenticationId PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingUsernameSaslPlain STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

[Paolo Patierno]

Hi Ismael,

first of all thanks for your reply.

So as far as I understood having a branch in the Kafka repo could be better for 
you as committer to validate small PRs from us and not a big one at the end, 
right ?


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience



From: isma...@gmail.com  on behalf of Ismael Juma 

Sent: Thursday, September 7, 2017 10:19 AM
To: dev@kafka.apache.org
Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new 
AdminClient

I would also add that it would be easier to review if there were smaller
PRs than one big PR. So, it may be worth thinking how progress could be
made more incrementally.

Ismael

On Thu, Sep 7, 2017 at 11:17 AM, Tom Bentley  wrote:

> I can't speak for the committers, but there's nothing to stop you
> submitting PRs against each others branches. It just needs you to agree
> which of you will host the integration branch. This would be pretty much
> exactly the same developer experience as of the branch was in the main
> Kafak repo AFAICS, except the committers wouldn't have to be involved with
> merging your PRs into your integration branch (which is probably a benefit
> to both you and them).
>
> On 7 September 2017 at 10:57, Paolo Patierno  wrote:
>
> > Hi committers,
> >
> >
> > as already asked by Viktor on the JIRA yesterday can you give us a
> > feedback/advice on how to move on that ? Thanks !
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperience
> >
> >
> > 
> > From: Paolo Patierno 
> > Sent: Thursday, August 17, 2017 3:18 PM
> > To: dev@kafka.apache.org
> > Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> > AdminClient
> >
> > As I started on working to develop the TopicCommand tool in Java (with
> > this PR), Andrey is working
> on
> > another one and he started to use some of my classes (mostly related to
> > handling command line arguments) just with copy/paste/modify so
> duplicating
> > the code.
> >
> > As mentioned in the JIRA discussion > apache.org/jira/browse/KAFKA-5723>, having a branch in the original repo
> > could help us to have personal forks with such branch, opening PRs
> against
> > this branch and having the common code there (syncing the code with
> forked
> > repos).
> >
> > In conclusion the problem is : having more people working on different
> > pieces of code which have some common code (still under development) not
> > yet available in the original repo.
> >
> > At least it's the idea we had but maybe the committers have a different
> > way to go in such situations. Any feedback is really appreciated :-)
> >
> >
> > Thanks,
> >
> > Paolo
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperience
> >
> >
> > 
> > From: Andrey Dyachkov 
> > Sent: Thursday, August 17, 2017 3:10 PM
> > To: dev@kafka.apache.org
> > Subject: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> > AdminClient
> >
> > Good day,
> > I am quite new here, my apologies for being kind of pushy.
> > We had discussion here https://issues.apache.org/jira/browse/KAFKA-5723,
> > that having separate branch for implementing the whole change
> > https://issues.apache.org/jira/browse/KAFKA-3268 would simplify dev
> > process. Could some of committers give a hand here?
> >
> > Also could you review pr https://github.com/apache/kafka/pull/3671 and
> add
> > me to contributors list in order to add myself to ticket assignee?
> >
> > Thank you!
> > --
> >
> > With great enthusiasm,
> > Andrey
> >
>

Re: [VOTE] KIP-188 - Add new metrics to support health checks

[Rajini Sivaram]

Hi all,

I have added one more metric to KIP-188 to show the current status of
broker's ZooKeeper connections. Please let me know if you have any concerns.

Hi Jun,

I was wondering which is a better group for FetchMessageConversionsPerSec,
now that we have MessageConversionsTimeMs at the request level. I don't
have a strong opinion either way, but at the moment, as a topic metric,
xxxMessageConversionsPerSec is along with MessagesInPerSec,
TotalFetchRequestsPerSec etc. which are all kind of related and are all
rate metrics. If we move it to the request level, we will
have MessageConversionsTimeMs and MessageConversionsPerSec together, but
will lose the topic grouping. Since all the other metrics at request level
are time histograms, perhaps it is better to leave MessageConversionsPerSec
along with the other topic rate metrics?

I had added a ZooKeeperClient wrapper for my initial PR, but I will rebase
on Onur's code when it is ready. Thank you!

Many thanks,

Rajini

On Wed, Sep 6, 2017 at 10:52 PM, Jun Rao  wrote:

> Hi, Raijini,
>
> Thanks for the KIP. +1. Just a minor comment.
>
> Since we only measure MessageConversionsTimeMs at the request type level,
> is it useful to collect the following metrics at the topic level?
>
> *MBean*:
> kafka.server:type=BrokerTopicMetrics,name=FetchMessageConversionsPerSec,
> topic=([-.\w]+)
>
> *MBean*:
> kafka.server:type=BrokerTopicMetrics,name=ProduceMessageConversionsPerSe
> c,topic=([-.\w]+)
>
>
> Also, for the ZK latency metric, Onur added a new ZookeeperClient wrapper
> and is in the middle of converting existing zkClient usage to the new
> wrapper. So, we probably want to add the latency metric in the new wrapper.
>
> Jun
>
> On Thu, Aug 24, 2017 at 10:50 AM, Rajini Sivaram 
> wrote:
>
> > Hi all,
> >
> > I would like to start the vote on KIP-188 that adds additional metrics to
> > support health checks for Kafka Ops. Details are here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 188+-+Add+new+metrics+to+support+health+checks
> >
> > Thank you,
> >
> > Rajini
> >
>

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

[Paolo Patierno]

Yes Tom that's true, we could chose one of us as the integration branch owner 
without involving committers but at same time it could be useful to involve 
committers for having the PRs evaluated step by step and not only at the end. 
For sure when committers are really busy for merging it's not good for us 
because we can't go forward with development.

Let's see what they say, maybe something like that already happened in the past 
...


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience



From: Tom Bentley 
Sent: Thursday, September 7, 2017 10:17 AM
To: dev@kafka.apache.org
Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new 
AdminClient

I can't speak for the committers, but there's nothing to stop you
submitting PRs against each others branches. It just needs you to agree
which of you will host the integration branch. This would be pretty much
exactly the same developer experience as of the branch was in the main
Kafak repo AFAICS, except the committers wouldn't have to be involved with
merging your PRs into your integration branch (which is probably a benefit
to both you and them).

On 7 September 2017 at 10:57, Paolo Patierno  wrote:

> Hi committers,
>
>
> as already asked by Viktor on the JIRA yesterday can you give us a
> feedback/advice on how to move on that ? Thanks !
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>
>
> 
> From: Paolo Patierno 
> Sent: Thursday, August 17, 2017 3:18 PM
> To: dev@kafka.apache.org
> Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> AdminClient
>
> As I started on working to develop the TopicCommand tool in Java (with
> this PR), Andrey is working on
> another one and he started to use some of my classes (mostly related to
> handling command line arguments) just with copy/paste/modify so duplicating
> the code.
>
> As mentioned in the JIRA discussion apache.org/jira/browse/KAFKA-5723>, having a branch in the original repo
> could help us to have personal forks with such branch, opening PRs against
> this branch and having the common code there (syncing the code with forked
> repos).
>
> In conclusion the problem is : having more people working on different
> pieces of code which have some common code (still under development) not
> yet available in the original repo.
>
> At least it's the idea we had but maybe the committers have a different
> way to go in such situations. Any feedback is really appreciated :-)
>
>
> Thanks,
>
> Paolo
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>
>
> 
> From: Andrey Dyachkov 
> Sent: Thursday, August 17, 2017 3:10 PM
> To: dev@kafka.apache.org
> Subject: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> AdminClient
>
> Good day,
> I am quite new here, my apologies for being kind of pushy.
> We had discussion here https://issues.apache.org/jira/browse/KAFKA-5723,
> that having separate branch for implementing the whole change
> https://issues.apache.org/jira/browse/KAFKA-3268 would simplify dev
> process. Could some of committers give a hand here?
>
> Also could you review pr https://github.com/apache/kafka/pull/3671 and add
> me to contributors list in order to add myself to ticket assignee?
>
> Thank you!
> --
>
> With great enthusiasm,
> Andrey
>

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

[Ismael Juma]

I would also add that it would be easier to review if there were smaller
PRs than one big PR. So, it may be worth thinking how progress could be
made more incrementally.

Ismael

On Thu, Sep 7, 2017 at 11:17 AM, Tom Bentley  wrote:

> I can't speak for the committers, but there's nothing to stop you
> submitting PRs against each others branches. It just needs you to agree
> which of you will host the integration branch. This would be pretty much
> exactly the same developer experience as of the branch was in the main
> Kafak repo AFAICS, except the committers wouldn't have to be involved with
> merging your PRs into your integration branch (which is probably a benefit
> to both you and them).
>
> On 7 September 2017 at 10:57, Paolo Patierno  wrote:
>
> > Hi committers,
> >
> >
> > as already asked by Viktor on the JIRA yesterday can you give us a
> > feedback/advice on how to move on that ? Thanks !
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperience
> >
> >
> > 
> > From: Paolo Patierno 
> > Sent: Thursday, August 17, 2017 3:18 PM
> > To: dev@kafka.apache.org
> > Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> > AdminClient
> >
> > As I started on working to develop the TopicCommand tool in Java (with
> > this PR), Andrey is working
> on
> > another one and he started to use some of my classes (mostly related to
> > handling command line arguments) just with copy/paste/modify so
> duplicating
> > the code.
> >
> > As mentioned in the JIRA discussion > apache.org/jira/browse/KAFKA-5723>, having a branch in the original repo
> > could help us to have personal forks with such branch, opening PRs
> against
> > this branch and having the common code there (syncing the code with
> forked
> > repos).
> >
> > In conclusion the problem is : having more people working on different
> > pieces of code which have some common code (still under development) not
> > yet available in the original repo.
> >
> > At least it's the idea we had but maybe the committers have a different
> > way to go in such situations. Any feedback is really appreciated :-)
> >
> >
> > Thanks,
> >
> > Paolo
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperience
> >
> >
> > 
> > From: Andrey Dyachkov 
> > Sent: Thursday, August 17, 2017 3:10 PM
> > To: dev@kafka.apache.org
> > Subject: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> > AdminClient
> >
> > Good day,
> > I am quite new here, my apologies for being kind of pushy.
> > We had discussion here https://issues.apache.org/jira/browse/KAFKA-5723,
> > that having separate branch for implementing the whole change
> > https://issues.apache.org/jira/browse/KAFKA-3268 would simplify dev
> > process. Could some of committers give a hand here?
> >
> > Also could you review pr https://github.com/apache/kafka/pull/3671 and
> add
> > me to contributors list in order to add myself to ticket assignee?
> >
> > Thank you!
> > --
> >
> > With great enthusiasm,
> > Andrey
> >
>

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

[Tom Bentley]

I can't speak for the committers, but there's nothing to stop you
submitting PRs against each others branches. It just needs you to agree
which of you will host the integration branch. This would be pretty much
exactly the same developer experience as of the branch was in the main
Kafak repo AFAICS, except the committers wouldn't have to be involved with
merging your PRs into your integration branch (which is probably a benefit
to both you and them).

On 7 September 2017 at 10:57, Paolo Patierno  wrote:

> Hi committers,
>
>
> as already asked by Viktor on the JIRA yesterday can you give us a
> feedback/advice on how to move on that ? Thanks !
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>
>
> 
> From: Paolo Patierno 
> Sent: Thursday, August 17, 2017 3:18 PM
> To: dev@kafka.apache.org
> Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> AdminClient
>
> As I started on working to develop the TopicCommand tool in Java (with
> this PR), Andrey is working on
> another one and he started to use some of my classes (mostly related to
> handling command line arguments) just with copy/paste/modify so duplicating
> the code.
>
> As mentioned in the JIRA discussion apache.org/jira/browse/KAFKA-5723>, having a branch in the original repo
> could help us to have personal forks with such branch, opening PRs against
> this branch and having the common code there (syncing the code with forked
> repos).
>
> In conclusion the problem is : having more people working on different
> pieces of code which have some common code (still under development) not
> yet available in the original repo.
>
> At least it's the idea we had but maybe the committers have a different
> way to go in such situations. Any feedback is really appreciated :-)
>
>
> Thanks,
>
> Paolo
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>
>
> 
> From: Andrey Dyachkov 
> Sent: Thursday, August 17, 2017 3:10 PM
> To: dev@kafka.apache.org
> Subject: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> AdminClient
>
> Good day,
> I am quite new here, my apologies for being kind of pushy.
> We had discussion here https://issues.apache.org/jira/browse/KAFKA-5723,
> that having separate branch for implementing the whole change
> https://issues.apache.org/jira/browse/KAFKA-3268 would simplify dev
> process. Could some of committers give a hand here?
>
> Also could you review pr https://github.com/apache/kafka/pull/3671 and add
> me to contributors list in order to add myself to ticket assignee?
>
> Thank you!
> --
>
> With great enthusiasm,
> Andrey
>