Build failed in Jenkins: kafka-trunk-jdk8 #1175

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-3209: KIP-66: single message transforms

[wangguoz] MINOR: make methods introduced in KAFKA-4490 consistent with KIP-100

[me] MINOR: avoid closing over both pre & post-transform record in

[ismael] MINOR: Remove unnecessary options in SCRAM test jaas config

[ismael] MINOR: Minor improvements in consumer close timeout handling

--
[...truncated 35097 lines...]

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[0] PASSED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[1] STARTED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[1] FAILED
java.lang.AssertionError: Condition not met within timeout 3. waiting 
for store count-by-key
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:259)
at 
org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shouldNotMakeStoreAvailableUntilAllStoresAvailable(QueryableStateIntegrationTest.java:501)

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[1] PASSED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > shouldThrowExceptionOverlappingPattern STARTED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > shouldThrowExceptionOverlappingPattern PASSED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > shouldThrowExceptionOverlappingTopic STARTED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > shouldThrowExceptionOverlappingTopic PASSED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > shouldThrowStreamsExceptionNoResetSpecified STARTED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > shouldThrowStreamsExceptionNoResetSpecified PASSED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > shouldOnlyReadRecordsWhereEarliestSpecified STARTED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > shouldOnlyReadRecordsWhereEarliestSpecified PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testShouldReadFromRegexAndNamedTopics STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testShouldReadFromRegexAndNamedTopics PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 

Build failed in Jenkins: kafka-trunk-jdk7 #1832

[Apache Jenkins Server]

See 

Changes:

[ismael] MINOR: Remove unnecessary options in SCRAM test jaas config

[ismael] MINOR: Minor improvements in consumer close timeout handling

--
[...truncated 18659 lines...]
org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCleanUpTaskStateDirectoriesThatAreNotCurrentlyLocked STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCleanUpTaskStateDirectoriesThatAreNotCurrentlyLocked PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCreateBaseDirectory STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCreateBaseDirectory PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldLockMulitpleTaskDirectories STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldLockMulitpleTaskDirectories PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCreateDirectoriesIfParentDoesntExist STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCreateDirectoriesIfParentDoesntExist PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldUnlockGlobalStateDirectory STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldUnlockGlobalStateDirectory PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldListAllTaskDirectories STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldListAllTaskDirectories PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldLockTaskStateDirectory STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldLockTaskStateDirectory PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldLockGlobalStateDirectory STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldLockGlobalStateDirectory PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCreateTaskStateDirectory STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCreateTaskStateDirectory PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldNotRemoveNonTaskDirectoriesAndFiles STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldNotRemoveNonTaskDirectoriesAndFiles PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldBeTrueIfAlreadyHoldsLock STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldBeTrueIfAlreadyHoldsLock PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldReleaseTaskStateDirectoryLock STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldReleaseTaskStateDirectoryLock PASSED

org.apache.kafka.streams.processor.internals.GlobalStateManagerImplTest > 
shouldThrowStreamsExceptionIfNoPartitionsFoundForStore STARTED

org.apache.kafka.streams.processor.internals.GlobalStateManagerImplTest > 
shouldThrowStreamsExceptionIfNoPartitionsFoundForStore PASSED

org.apache.kafka.streams.processor.internals.GlobalStateManagerImplTest > 
shouldNotCloseStoresIfCloseAlreadyCalled STARTED

org.apache.kafka.streams.processor.internals.GlobalStateManagerImplTest > 
shouldNotCloseStoresIfCloseAlreadyCalled PASSED

org.apache.kafka.streams.processor.internals.GlobalStateManagerImplTest 

Build failed in Jenkins: kafka-trunk-jdk7 #1831

[Apache Jenkins Server]

See 

Changes:

[me] MINOR: avoid closing over both pre & post-transform record in

--
[...truncated 18646 lines...]
org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCleanUpTaskStateDirectoriesThatAreNotCurrentlyLocked STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCleanUpTaskStateDirectoriesThatAreNotCurrentlyLocked PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCreateBaseDirectory STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCreateBaseDirectory PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldLockMulitpleTaskDirectories STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldLockMulitpleTaskDirectories PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCreateDirectoriesIfParentDoesntExist STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCreateDirectoriesIfParentDoesntExist PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldUnlockGlobalStateDirectory STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldUnlockGlobalStateDirectory PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldListAllTaskDirectories STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldListAllTaskDirectories PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldLockTaskStateDirectory STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldLockTaskStateDirectory PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldLockGlobalStateDirectory STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldLockGlobalStateDirectory PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCreateTaskStateDirectory STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldCreateTaskStateDirectory PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldNotRemoveNonTaskDirectoriesAndFiles STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldNotRemoveNonTaskDirectoriesAndFiles PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldBeTrueIfAlreadyHoldsLock STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldBeTrueIfAlreadyHoldsLock PASSED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldReleaseTaskStateDirectoryLock STARTED

org.apache.kafka.streams.processor.internals.StateDirectoryTest > 
shouldReleaseTaskStateDirectoryLock PASSED

org.apache.kafka.streams.processor.internals.GlobalStateManagerImplTest > 
shouldThrowStreamsExceptionIfNoPartitionsFoundForStore STARTED

org.apache.kafka.streams.processor.internals.GlobalStateManagerImplTest > 
shouldThrowStreamsExceptionIfNoPartitionsFoundForStore PASSED

org.apache.kafka.streams.processor.internals.GlobalStateManagerImplTest > 
shouldNotCloseStoresIfCloseAlreadyCalled STARTED

org.apache.kafka.streams.processor.internals.GlobalStateManagerImplTest > 
shouldNotCloseStoresIfCloseAlreadyCalled PASSED

org.apache.kafka.streams.processor.internals.GlobalStateManagerImplTest > 
shouldThrowProcessorStateStoreExceptionIfStoreFlushFailed STARTED

Jenkins build is back to normal : kafka-trunk-jdk8 #1174

[Apache Jenkins Server]

See 

[GitHub] kafka pull request #2348: MINOR: Minor improvements in consumer close timeou...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2348


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[Ismael Juma]

Thanks for the KIP. Just wanted to quickly say that it's great to see
proposals for improving JBOD (KIP-113 too). More feedback soon, hopefully.

Ismael

On Thu, Jan 12, 2017 at 6:46 PM, Dong Lin  wrote:

> Hi all,
>
> We created KIP-112: Handle disk failure for JBOD. Please find the KIP wiki
> in the link https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 112%3A+Handle+disk+failure+for+JBOD.
>
> This KIP is related to KIP-113
>  113%3A+Support+replicas+movement+between+log+directories>:
> Support replicas movement between log directories. They are needed in order
> to support JBOD in Kafka. Please help review the KIP. You feedback is
> appreciated!
>
> Thanks,
> Dong
>

[GitHub] kafka pull request #2353: MINOR: Remove unnecessary options in SCRAM test ja...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2353


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2365: MINOR: avoid closing over both pre & post-transfor...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2365


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2365: MINOR: avoid closing over both pre & post-transfor...

[shikhar]

GitHub user shikhar opened a pull request:

https://github.com/apache/kafka/pull/2365

MINOR: avoid closing over both pre & post-transform record in 
WorkerSourceTask

Followup to #2299 for KAFKA-3209

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shikhar/kafka 2299-followup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2365.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2365


commit b0abf743d0329bbbc9620b1b0ef6acd4b3b035b3
Author: Shikhar Bhushan 
Date:   2017-01-13T00:32:11Z

MINOR: avoid closing over both pre & post-transform record in 
WorkerSourceTask

Followup to #2299 for KAFKA-3209




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2363: MINOR: make methods introduced in KAFKA-4490 consi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2363


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2299: KAFKA-3209: KIP-66: single message transforms

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2299


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2364: MINOR: fix JavaDoc

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2364


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2364: MINOR: fix JavaDoc

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/2364

MINOR: fix JavaDoc



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka hotfix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2364.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2364


commit b078607ae184f4c241a39ee8eaf816eec6cfef7e
Author: Matthias J. Sax 
Date:   2017-01-12T22:43:31Z

MINOR: fix JavaDoc




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2363: MINOR: make methods introduced in KAFKA-4490 consi...

[xvrl]

GitHub user xvrl opened a pull request:

https://github.com/apache/kafka/pull/2363

MINOR: make methods introduced in KAFKA-4490 consistent with KIP-100

and remove some unnecessary @SuppressWarnings annotations

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xvrl/kafka kip-100-followup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2363.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2363


commit 0ce3b71fd9b557989729a4497029b17cfea0e6a5
Author: Xavier Léauté 
Date:   2017-01-12T22:33:54Z

make methods introduced in KAFKA-4490 consistent with KIP-100

and remove some unnecessary @SuppressWarnings annotations




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2362: HOTFIX: Added another broker to smoke test

[enothereska]

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/2362

HOTFIX: Added another broker to smoke test



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka hotfix-smoke-test-2-brokers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2362.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2362


commit 46d2af77f737818b23d5893a07f27f2b715fd0f2
Author: Eno Thereska 
Date:   2017-01-12T21:10:03Z

Added another broker




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

UNSUBSCRIBE PLEASE

[williamtellme123]

us...@kafka.apache.org;users-unsubscr...@kafka.apache.org;
users_unsubscr...@kafka.apache.org;
dev@kafka.apache.org; dev-unsubscr...@kafka.apache.org;
dev_unsubscr...@kafka.apache.org
-Original Message-
From: Raj Tanneru [mailto:rtann...@fanatics.com] 
Sent: Saturday, May 7, 2016 6:46 PM
To: us...@kafka.apache.org
Cc: dev@kafka.apache.org
Subject: Re: KAFKA-3112

Thanks Ismael and Tao. Appreciate it.

Sent from my iPhone

> On May 7, 2016, at 1:14 AM, Ismael Juma  wrote:
>
> Hi Raj and Tao,
>
> I just merged the KAFKA-3112 PR, so this issue will be fixed in 0.10.0.0.
>
> Thanks,
> Ismael
>
>> On Fri, May 6, 2016 at 7:47 PM, tao xiao  wrote:
>>
>> KAFKA-2657 is unresolved so you can safely assume it hasn't been fixed
yet.
>>
>>> On Fri, 6 May 2016 at 07:38 Raj Tanneru  wrote:
>>>
>>> Yeah it is a duplicate of KAFKA-2657. The question is how to check / 
>>> know if it is merged to 0.9.0.1 release. What are the options that I 
>>> have if I need this fix. How can I get patch for this on 0.8.2.1?
>>>
>>> Sent from my iPhone
>>>
 On May 6, 2016, at 12:06 AM, tao xiao  wrote:

 It said this is a duplication. This is the
 https://issues.apache.org/jira/browse/KAFKA-2657 that KAKFA-3112
>>> duplicates
 to.

> On Thu, 5 May 2016 at 22:13 Raj Tanneru 
>> wrote:
>
>
> Hi All,
> Does anyone know if KAFKA-3112 is merged to 0.9.0.1? Is there a 
> place
>> to
> check which version has this fix? Jira doesn't show fix versions.
>
> https://issues.apache.org/jira/browse/KAFKA-3112
>
>
> Thanks,
> Raj Tanneru
> Information contained in this e-mail message is confidential. This
>>> e-mail
> message is intended only for the personal use of the recipient(s)
>> named
> above. If you are not an intended recipient, do not read, 
> distribute
>> or
> reproduce this transmission (including any attachments). If you 
> have received this email in error, please immediately notify the 
> sender by
>>> email
> reply and delete the original message.
>>> Information contained in this e-mail message is confidential. This 
>>> e-mail message is intended only for the personal use of the 
>>> recipient(s) named above. If you are not an intended recipient, do 
>>> not read, distribute or reproduce this transmission (including any 
>>> attachments). If you have received this email in error, please 
>>> immediately notify the sender by
>> email
>>> reply and delete the original message.
>>
Information contained in this e-mail message is confidential. This e-mail
message is intended only for the personal use of the recipient(s) named
above. If you are not an intended recipient, do not read, distribute or
reproduce this transmission (including any attachments). If you have
received this email in error, please immediately notify the sender by email
reply and delete the original message.

Build failed in Jenkins: kafka-trunk-jdk8 #1173

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-4603: Fixed the argument of shell in doc

[cshapi] MINOR: Fix error in design docs

[wangguoz] KAFKA-4490: Add Global Table support to Kafka Streams

--
[...truncated 15470 lines...]
org.apache.kafka.common.security.scram.ScramCredentialUtilsTest > 
generateCredential PASSED

org.apache.kafka.common.security.scram.ScramCredentialUtilsTest > 
extraneousFields STARTED

org.apache.kafka.common.security.scram.ScramCredentialUtilsTest > 
extraneousFields PASSED

org.apache.kafka.common.security.scram.ScramCredentialUtilsTest > 
scramCredentialCache STARTED

org.apache.kafka.common.security.scram.ScramCredentialUtilsTest > 
scramCredentialCache PASSED

org.apache.kafka.common.security.scram.ScramCredentialUtilsTest > 
invalidCredential STARTED

org.apache.kafka.common.security.scram.ScramCredentialUtilsTest > 
invalidCredential PASSED

org.apache.kafka.common.security.scram.ScramFormatterTest > rfc7677Example 
STARTED

org.apache.kafka.common.security.scram.ScramFormatterTest > rfc7677Example 
PASSED

org.apache.kafka.common.security.scram.ScramFormatterTest > saslName STARTED

org.apache.kafka.common.security.scram.ScramFormatterTest > saslName PASSED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
validClientFirstMessage STARTED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
validClientFirstMessage PASSED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
invalidClientFinalMessage STARTED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
invalidClientFinalMessage PASSED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
validServerFirstMessage STARTED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
validServerFirstMessage PASSED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
invalidServerFinalMessage STARTED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
invalidServerFinalMessage PASSED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
invalidClientFirstMessage STARTED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
invalidClientFirstMessage PASSED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
validClientFinalMessage STARTED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
validClientFinalMessage PASSED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
invalidServerFirstMessage STARTED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
invalidServerFirstMessage PASSED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
validServerFinalMessage STARTED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
validServerFinalMessage PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > testClientMode STARTED

org.apache.kafka.common.security.ssl.SslFactoryTest > testClientMode PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryConfiguration STARTED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryConfiguration PASSED

org.apache.kafka.common.security.kerberos.KerberosNameTest > testParse STARTED

org.apache.kafka.common.security.kerberos.KerberosNameTest > testParse PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testPrincipalNameCanContainSeparator STARTED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testPrincipalNameCanContainSeparator PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testEqualsAndHashCode STARTED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testEqualsAndHashCode PASSED

org.apache.kafka.common.security.JaasUtilsTest > testMissingOptionValue STARTED

org.apache.kafka.common.security.JaasUtilsTest > testMissingOptionValue PASSED

org.apache.kafka.common.security.JaasUtilsTest > testSingleOption STARTED

org.apache.kafka.common.security.JaasUtilsTest > testSingleOption PASSED

org.apache.kafka.common.security.JaasUtilsTest > testNumericOptionWithoutQuotes 
STARTED

org.apache.kafka.common.security.JaasUtilsTest > testNumericOptionWithoutQuotes 
PASSED

org.apache.kafka.common.security.JaasUtilsTest > testConfigNoOptions STARTED

org.apache.kafka.common.security.JaasUtilsTest > testConfigNoOptions PASSED

org.apache.kafka.common.security.JaasUtilsTest > testNumericOptionWithQuotes 
STARTED

org.apache.kafka.common.security.JaasUtilsTest > testNumericOptionWithQuotes 
PASSED

org.apache.kafka.common.security.JaasUtilsTest > testQuotedOptionValue STARTED

org.apache.kafka.common.security.JaasUtilsTest > testQuotedOptionValue PASSED

org.apache.kafka.common.security.JaasUtilsTest > testMissingLoginModule STARTED

org.apache.kafka.common.security.JaasUtilsTest > testMissingLoginModule PASSED

org.apache.kafka.common.security.JaasUtilsTest > testMissingSemicolon STARTED

org.apache.kafka.common.security.JaasUtilsTest > testMissingSemicolon PASSED

[GitHub] kafka pull request #2361: KAFKA-4591: Create Topic Policy (KIP-108)

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/2361

KAFKA-4591: Create Topic Policy (KIP-108)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka kafka-4591-create-topic-policy

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2361.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2361


commit 89736a747d8997118578a22f8fcc9ceacd1dad7e
Author: Ismael Juma 
Date:   2017-01-03T01:04:21Z

KAFKA-4591: Create Topic Policy (KIP-108)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2360: KAFKA-3452 Follow-up: Refactoring StateStore hiera...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/2360

KAFKA-3452 Follow-up: Refactoring StateStore hierarchies

This is a follow up of https://github.com/apache/kafka/pull/2166 - 
refactoring the store hierarchies as requested

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka state-store-refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2360.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2360


commit 79f59911ee0eb74b0e653c920071b9bd5e4a2239
Author: Damian Guy 
Date:   2017-01-11T15:35:11Z

refactor store hierarchies




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2244: KAFKA-4490: Add Global Table support to Kafka Stre...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2244


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2357: Fix error in design docs

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2357


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2322: KAFKA-4603 the argument of shell in doc wrong and ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2322


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

[Mayuresh Gharat]

Hi Ismael,

Fair point. I will update it.

Thanks,

Mayuresh

On Thu, Jan 12, 2017 at 11:07 AM, Ismael Juma  wrote:

> Hi Mayuresh,
>
> Thanks for the KIP. A quick comment before I do a more detailed analysis,
> the KIP says:
>
> `This KIP is a pure addition to existing functionality and does not include
> any backward incompatible changes.`
>
> However, the KIP is proposing the addition of a method to the
> PrincipalBuilder pluggable interface, which is not a compatible change.
> Existing implementations would no longer compile, for example. It would be
> good to make this clear in the KIP.
>
> Ismael
>
> On Thu, Jan 12, 2017 at 5:44 PM, Mayuresh Gharat <
> gharatmayures...@gmail.com
> > wrote:
>
> > Hi all.
> >
> > We created KIP-111 to propose that Kafka should preserve the Principal
> > generated by the PrincipalBuilder while processing the request received
> on
> > socket channel, on the broker.
> >
> > Please find the KIP wiki in the link
> > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=67638388.
> > We would love to hear your comments and suggestions.
> >
> >
> > Thanks,
> >
> > Mayuresh
> >
>



-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

Re: [DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

[Ismael Juma]

Hi Mayuresh,

Thanks for the KIP. A quick comment before I do a more detailed analysis,
the KIP says:

`This KIP is a pure addition to existing functionality and does not include
any backward incompatible changes.`

However, the KIP is proposing the addition of a method to the
PrincipalBuilder pluggable interface, which is not a compatible change.
Existing implementations would no longer compile, for example. It would be
good to make this clear in the KIP.

Ismael

On Thu, Jan 12, 2017 at 5:44 PM, Mayuresh Gharat  wrote:

> Hi all.
>
> We created KIP-111 to propose that Kafka should preserve the Principal
> generated by the PrincipalBuilder while processing the request received on
> socket channel, on the broker.
>
> Please find the KIP wiki in the link
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=67638388.
> We would love to hear your comments and suggestions.
>
>
> Thanks,
>
> Mayuresh
>

[DISCUSS] KIP-113: Support replicas movement between log directories

[Dong Lin]

Hi all,

We created KIP-113: Support replicas movement between log directories.
Please find the KIP wiki in the link
*https://cwiki.apache.org/confluence/display/KAFKA/KIP-113%3A+Support+replicas+movement+between+log+directories
.*

This KIP is related to KIP-112
:
Handle disk failure for JBOD. They are needed in order to support JBOD in
Kafka. Please help review the KIP. You feedback is appreciated!

Thanks,
Dong

[DISCUSS] KIP-112: Handle disk failure for JBOD

[Dong Lin]

Hi all,

We created KIP-112: Handle disk failure for JBOD. Please find the KIP wiki
in the link https://cwiki.apache.org/confluence/display/KAFKA/KIP-
112%3A+Handle+disk+failure+for+JBOD.

This KIP is related to KIP-113
:
Support replicas movement between log directories. They are needed in order
to support JBOD in Kafka. Please help review the KIP. You feedback is
appreciated!

Thanks,
Dong

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

[Joel Koshy]

+1

(for the record, I favor the rejected alternative of not awaiting low
watermarks to go past the purge offset. I realize it offers a weaker
guarantee but it is still very useful, easier to implement, slightly
simpler API (no need to return a future) and you can still get access to
the current low watermark via a fetch request; although it would be weird
to include the low watermark on the purge response in this variation)

On Wed, Jan 11, 2017 at 1:01 PM, Dong Lin  wrote:

> Hi all,
>
> It seems that there is no further concern with the KIP-107. At this point
> we would like to start the voting process. The KIP can be found at
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-107
> %3A+Add+purgeDataBefore%28%29+API+in+AdminClient.
>
> Thanks,
> Dong
>

Re: [ANNOUNCE] New committer: Grant Henke

[Joel Koshy]

Hey Grant - congrats!

On Thu, Jan 12, 2017 at 10:00 AM, Neha Narkhede  wrote:

> Congratulations, Grant. Well deserved!
>
> On Thu, Jan 12, 2017 at 7:51 AM Grant Henke  wrote:
>
> > Thanks everyone!
> >
> > On Thu, Jan 12, 2017 at 2:58 AM, Damian Guy 
> wrote:
> >
> > > Congratulations!
> > >
> > > On Thu, 12 Jan 2017 at 03:35 Jun Rao  wrote:
> > >
> > > > Grant,
> > > >
> > > > Thanks for all your contribution! Congratulations!
> > > >
> > > > Jun
> > > >
> > > > On Wed, Jan 11, 2017 at 2:51 PM, Gwen Shapira 
> > wrote:
> > > >
> > > > > The PMC for Apache Kafka has invited Grant Henke to join as a
> > > > > committer and we are pleased to announce that he has accepted!
> > > > >
> > > > > Grant contributed 88 patches, 90 code reviews, countless great
> > > > > comments on discussions, a much-needed cleanup to our protocol and
> > the
> > > > > on-going and critical work on the Admin protocol. Throughout this,
> he
> > > > > displayed great technical judgment, high-quality work and
> willingness
> > > > > to contribute where needed to make Apache Kafka awesome.
> > > > >
> > > > > Thank you for your contributions, Grant :)
> > > > >
> > > > > --
> > > > > Gwen Shapira
> > > > > Product Manager | Confluent
> > > > > 650.450.2760 <(650)%20450-2760> <(650)%20450-2760> | @gwenshap
> > > > > Follow us: Twitter | blog
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >
> --
> Thanks,
> Neha
>

[GitHub] kafka pull request #2359: KAFKA-3452: follow up - fix state store restoratio...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/2359

KAFKA-3452: follow up - fix state store restoration for session and window 
stores

The refactoring of the stores in https://github.com/apache/kafka/pull/2166 
introduced `ChangeLoggingSegmentedBytesStore`, thus removing the change logging 
from the `RocksDBWindowStore` etc. However, the inner most store still needs to 
know whether or not logging is enabled such that it can register correctly with 
`ProcessorContext` and enable StateStore restoration

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka hot-fix-store-logging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2359.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2359


commit 0416c1dccf0b36d7614ccd58d478415176577cb3
Author: Damian Guy 
Date:   2017-01-12T17:59:57Z

fix changelogging in session and window stores




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [ANNOUNCE] New committer: Grant Henke

[Neha Narkhede]

Congratulations, Grant. Well deserved!

On Thu, Jan 12, 2017 at 7:51 AM Grant Henke  wrote:

> Thanks everyone!
>
> On Thu, Jan 12, 2017 at 2:58 AM, Damian Guy  wrote:
>
> > Congratulations!
> >
> > On Thu, 12 Jan 2017 at 03:35 Jun Rao  wrote:
> >
> > > Grant,
> > >
> > > Thanks for all your contribution! Congratulations!
> > >
> > > Jun
> > >
> > > On Wed, Jan 11, 2017 at 2:51 PM, Gwen Shapira 
> wrote:
> > >
> > > > The PMC for Apache Kafka has invited Grant Henke to join as a
> > > > committer and we are pleased to announce that he has accepted!
> > > >
> > > > Grant contributed 88 patches, 90 code reviews, countless great
> > > > comments on discussions, a much-needed cleanup to our protocol and
> the
> > > > on-going and critical work on the Admin protocol. Throughout this, he
> > > > displayed great technical judgment, high-quality work and willingness
> > > > to contribute where needed to make Apache Kafka awesome.
> > > >
> > > > Thank you for your contributions, Grant :)
> > > >
> > > > --
> > > > Gwen Shapira
> > > > Product Manager | Confluent
> > > > 650.450.2760 <(650)%20450-2760> <(650)%20450-2760> | @gwenshap
> > > > Follow us: Twitter | blog
> > > >
> > >
> >
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>
-- 
Thanks,
Neha

[GitHub] kafka pull request #2358: WIP -- DO NOT MERGE -- KAFKA-4222: Transient failu...

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/2358

WIP -- DO NOT MERGE -- KAFKA-4222: Transient failure in 
QueryableStateIntegrationTest.queryOnRebalance



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka kafka-4222-instable-iq-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2358.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2358


commit a7e6e149f74576a0ed60cd3e4836a678122c9a17
Author: Matthias J. Sax 
Date:   2017-01-12T17:55:10Z

WIP -- DO NOT MERGE -- KAFKA-4222: Transient failure in 
QueryableStateIntegrationTest.queryOnRebalance




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

[Mayuresh Gharat]

+1 (non-binding).

Thanks,

Mayuresh

On Wed, Jan 11, 2017 at 10:11 PM, radai  wrote:

> LGTM, +1
>
> On Wed, Jan 11, 2017 at 1:01 PM, Dong Lin  wrote:
>
> > Hi all,
> >
> > It seems that there is no further concern with the KIP-107. At this point
> > we would like to start the voting process. The KIP can be found at
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-107
> > %3A+Add+purgeDataBefore%28%29+API+in+AdminClient.
> >
> > Thanks,
> > Dong
> >
>



-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

[DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

[Mayuresh Gharat]

Hi all.

We created KIP-111 to propose that Kafka should preserve the Principal
generated by the PrincipalBuilder while processing the request received on
socket channel, on the broker.

Please find the KIP wiki in the link
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=67638388.
We would love to hear your comments and suggestions.


Thanks,

Mayuresh

Re: [VOTE] KIP-107: Add purgeDataBefore() API in AdminClient

[Mayuresh Gharat]

+1 (non-binding)

Thanks,

Mayuresh

On Wed, Jan 11, 2017 at 1:03 PM, Dong Lin  wrote:

> Sorry for the duplicated email. It seems that gmail will put the voting
> email in this thread if I simply replace DISCUSS with VOTE in the subject.
>
> On Wed, Jan 11, 2017 at 12:57 PM, Dong Lin  wrote:
>
> > Hi all,
> >
> > It seems that there is no further concern with the KIP-107. At this point
> > we would like to start the voting process. The KIP can be found at
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-107
> > %3A+Add+purgeDataBefore%28%29+API+in+AdminClient.
> >
> > Thanks,
> > Dong
> >
>



-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

[GitHub] kafka pull request #2357: Fix error in design docs

[dasl-]

GitHub user dasl- opened a pull request:

https://github.com/apache/kafka/pull/2357

Fix error in design docs



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dasl-/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2357.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2357


commit 7e6468005e5c282b0a5310b16c56fa65e428e32d
Author: dasl 
Date:   2017-01-12T17:14:14Z

Fix error in design docs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [ANNOUNCE] New committer: Grant Henke

[Grant Henke]

Thanks everyone!

On Thu, Jan 12, 2017 at 2:58 AM, Damian Guy  wrote:

> Congratulations!
>
> On Thu, 12 Jan 2017 at 03:35 Jun Rao  wrote:
>
> > Grant,
> >
> > Thanks for all your contribution! Congratulations!
> >
> > Jun
> >
> > On Wed, Jan 11, 2017 at 2:51 PM, Gwen Shapira  wrote:
> >
> > > The PMC for Apache Kafka has invited Grant Henke to join as a
> > > committer and we are pleased to announce that he has accepted!
> > >
> > > Grant contributed 88 patches, 90 code reviews, countless great
> > > comments on discussions, a much-needed cleanup to our protocol and the
> > > on-going and critical work on the Admin protocol. Throughout this, he
> > > displayed great technical judgment, high-quality work and willingness
> > > to contribute where needed to make Apache Kafka awesome.
> > >
> > > Thank you for your contributions, Grant :)
> > >
> > > --
> > > Gwen Shapira
> > > Product Manager | Confluent
> > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> > > Follow us: Twitter | blog
> > >
> >
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

[jira] [Commented] (KAFKA-4581) Fail gracefully if multiple login modules are specified in sasl.jaas.config

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821237#comment-15821237
 ] 

ASF GitHub Bot commented on KAFKA-4581:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/2356

KAFKA-4581: Fail early if multiple client login modules in sasl.jaas.config

Validate and fail client connection if multiple login modules are specified 
in sasl.jaas.config to avoid harder-to-debug authentication failures later on.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4581

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2356.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2356


commit 1166f3c7a811f5546de44fc68407d87aae69a383
Author: Rajini Sivaram 
Date:   2017-01-12T15:10:14Z

KAFKA-4581: Fail early if multiple client login modules specified in
sasl.jaas.config




> Fail gracefully if multiple login modules are specified in sasl.jaas.config
> ---
>
> Key: KAFKA-4581
> URL: https://issues.apache.org/jira/browse/KAFKA-4581
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.10.2.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.10.2.0
>
>
> Validate config and throw meaningful exception if multiple login modules are 
> specified for client in sasl.jaas.config property.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2356: KAFKA-4581: Fail early if multiple client login mo...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/2356

KAFKA-4581: Fail early if multiple client login modules in sasl.jaas.config

Validate and fail client connection if multiple login modules are specified 
in sasl.jaas.config to avoid harder-to-debug authentication failures later on.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4581

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2356.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2356


commit 1166f3c7a811f5546de44fc68407d87aae69a383
Author: Rajini Sivaram 
Date:   2017-01-12T15:10:14Z

KAFKA-4581: Fail early if multiple client login modules specified in
sasl.jaas.config




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Work started] (KAFKA-4604) Gradle Eclipse plugin creates projects for non project subfolders

[Dhwani Katagade (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-4604 started by Dhwani Katagade.
--
> Gradle Eclipse plugin creates projects for non project subfolders
> -
>
> Key: KAFKA-4604
> URL: https://issues.apache.org/jira/browse/KAFKA-4604
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.11.0.0
> Environment: Tried with Gradle 3.2.1 and Eclipse Neon
>Reporter: Dhwani Katagade
>Assignee: Dhwani Katagade
>Priority: Minor
>  Labels: build, easyfix, patch
>
> Running the command *./gradlew eclipse* generates .project and .classpath 
> files for all projects. But it also generates these files for the root 
> project folder and the connect subfolder that holds the 4 connector projects 
> even though these folders are not actual project folders.
> The unnecessary connect project is benign, but the unnecessary kafka project 
> created for the root folder has a side effect. The root folder has a bin 
> directory that holds some scripts. When a _Clean all projects_ is done in 
> Eclipse, it cleans up the scripts in the bin directory. These have to be 
> restored by running *git checkout \-\- *. This same could become a 
> problem for the connect project as well if tomorrow we place some files under 
> connect/bin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4617) gradle-generated core eclipse project has incorrect source folder structure

[Edoardo Comar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edoardo Comar updated KAFKA-4617:
-
Description: 
The gradle-generated Eclipse Scala project for Kafka core has a 
classpath defined as :
{code:xml}



{code}
because of how the source files are for tests are structured, code navigation / 
running unit tests fails. The correct structure should be instead :
{code:xml}






{code}

Moreover, the classpath included as libraries core/build/test and 
core/build/resources
which should not be there as the eclipse classes are not generated under build


  was:
The gradle-generated Eclipse Scala project for Kafka core has a 
classpath defined as :
{code:xml}



{code}
because of how the source files are for tests are structured, code navigation / 
running unit tests fails. The correct structure should be instead :
{code:xml}






{code}


> gradle-generated core eclipse project has incorrect source folder structure
> ---
>
> Key: KAFKA-4617
> URL: https://issues.apache.org/jira/browse/KAFKA-4617
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Reporter: Edoardo Comar
>Priority: Minor
>
> The gradle-generated Eclipse Scala project for Kafka core has a 
> classpath defined as :
> {code:xml}
>   
>   
>   
> {code}
> because of how the source files are for tests are structured, code navigation 
> / running unit tests fails. The correct structure should be instead :
> {code:xml}
>   
>path="src/test/scala"/>
>   
>   
>   
>   
> {code}
> Moreover, the classpath included as libraries core/build/test and 
> core/build/resources
> which should not be there as the eclipse classes are not generated under build



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4565) Separation of Internal and External traffic (KIP-103)

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4565:
---
Status: Patch Available  (was: In Progress)

> Separation of Internal and External traffic (KIP-103)
> -
>
> Key: KAFKA-4565
> URL: https://issues.apache.org/jira/browse/KAFKA-4565
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>  Labels: kip
> Fix For: 0.10.2.0
>
>
> During the 0.9.0.0 release cycle, support for multiple listeners per broker 
> was introduced (KAFKA-1809). Each listener is associated with a security 
> protocol, ip/host and port. When combined with the advertised listeners 
> mechanism, there is a fair amount of flexibility with one limitation: at most 
> one listener per security protocol in each of the two configs (listeners and 
> advertised.listeners).
> In some environments, one may want to differentiate between external clients, 
> internal clients and replication traffic independently of the security 
> protocol for cost, performance and security reasons. See the KIP for more 
> details: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4590) Add system test for SASL/SCRAM

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821084#comment-15821084
 ] 

ASF GitHub Bot commented on KAFKA-4590:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/2355

KAFKA-4590: SASL/SCRAM system tests

Runs sanity test and one replication test using SASL/SCRAM.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4590

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2355.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2355


commit 1d09e85150023fc1389bf897c8cac618db70acb8
Author: Rajini Sivaram 
Date:   2017-01-12T13:20:59Z

KAFKA-4590: SASL/SCRAM system tests




> Add system test for SASL/SCRAM
> --
>
> Key: KAFKA-4590
> URL: https://issues.apache.org/jira/browse/KAFKA-4590
> Project: Kafka
>  Issue Type: Task
>  Components: system tests
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.10.2.0
>
>
> Add system tests for SASL/SCRAM. This corresponds to 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-84%3A+Support+SASL+SCRAM+mechanisms,
>  KAFKA-3751.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2355: KAFKA-4590: SASL/SCRAM system tests

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/2355

KAFKA-4590: SASL/SCRAM system tests

Runs sanity test and one replication test using SASL/SCRAM.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4590

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2355.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2355


commit 1d09e85150023fc1389bf897c8cac618db70acb8
Author: Rajini Sivaram 
Date:   2017-01-12T13:20:59Z

KAFKA-4590: SASL/SCRAM system tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4565) Separation of Internal and External traffic (KIP-103)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821082#comment-15821082
 ] 

ASF GitHub Bot commented on KAFKA-4565:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/2354

KAFKA-4565: Separation of Internal and External traffic (KIP-103)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-4565-separation-of-internal-and-external-traffic

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2354.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2354


commit 8c5dc93e2ccea2c4c65e6ce28f22adf0dd9c6be8
Author: Ismael Juma 
Date:   2017-01-12T14:11:20Z

KAFKA-4565: Separation of Internal and External traffic (KIP-103)




> Separation of Internal and External traffic (KIP-103)
> -
>
> Key: KAFKA-4565
> URL: https://issues.apache.org/jira/browse/KAFKA-4565
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>  Labels: kip
> Fix For: 0.10.2.0
>
>
> During the 0.9.0.0 release cycle, support for multiple listeners per broker 
> was introduced (KAFKA-1809). Each listener is associated with a security 
> protocol, ip/host and port. When combined with the advertised listeners 
> mechanism, there is a fair amount of flexibility with one limitation: at most 
> one listener per security protocol in each of the two configs (listeners and 
> advertised.listeners).
> In some environments, one may want to differentiate between external clients, 
> internal clients and replication traffic independently of the security 
> protocol for cost, performance and security reasons. See the KIP for more 
> details: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2354: KAFKA-4565: Separation of Internal and External tr...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/2354

KAFKA-4565: Separation of Internal and External traffic (KIP-103)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-4565-separation-of-internal-and-external-traffic

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2354.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2354


commit 8c5dc93e2ccea2c4c65e6ce28f22adf0dd9c6be8
Author: Ismael Juma 
Date:   2017-01-12T14:11:20Z

KAFKA-4565: Separation of Internal and External traffic (KIP-103)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3745) Consider adding join key to ValueJoiner interface

[Michael Noll (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821054#comment-15821054
 ] 

Michael Noll commented on KAFKA-3745:
-

I think [~mjsax] meant to say "not a big deal" (rather than "big dead") :-P

> Consider adding join key to ValueJoiner interface
> -
>
> Key: KAFKA-3745
> URL: https://issues.apache.org/jira/browse/KAFKA-3745
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Greg Fodor
>Assignee: Sreepathi Prasanna
>Priority: Minor
>  Labels: api, needs-kip, newbie
>
> In working with Kafka Stream joining, it's sometimes the case that a join key 
> is not actually present in the values of the joins themselves (if, for 
> example, a previous transform generated an ephemeral join key.) In such 
> cases, the actual key of the join is not available in the ValueJoiner 
> implementation to be used to construct the final joined value. This can be 
> worked around by explicitly threading the join key into the value if needed, 
> but it seems like extending the interface to pass the join key along as well 
> would be helpful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4616) Message loss is seen when kafka-producer-perf-test.sh is running and any broker restarted in middle

[sandeep kumar singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sandeep kumar singh updated KAFKA-4616:
---
Summary: Message loss is seen when kafka-producer-perf-test.sh is running 
and any broker restarted in middle  (was: Message log is seen when 
kafka-producer-perf-test.sh is running and any broker restarted in middle 
in-between )

> Message loss is seen when kafka-producer-perf-test.sh is running and any 
> broker restarted in middle
> ---
>
> Key: KAFKA-4616
> URL: https://issues.apache.org/jira/browse/KAFKA-4616
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
> Environment: Apache mesos
>Reporter: sandeep kumar singh
>
> if any broker is restarted while kafka-producer-perf-test.sh command is 
> running, we see message loss.
> commands i run:
> **perf command:
> $ bin/kafka-producer-perf-test.sh --num-records 10 --record-size 4096  
> --throughput 1000 --topic test3R3P3 --producer-props 
> bootstrap.servers=x.x.x.x:,x.x.x.x:,x.x.x.x:
> I am  sending 10 messages of each having size 4096
> error thrown by perf command:
> 4944 records sent, 988.6 records/sec (3.86 MB/sec), 31.5 ms avg latency, 
> 433.0 max latency.
> 5061 records sent, 1012.0 records/sec (3.95 MB/sec), 67.7 ms avg latency, 
> 798.0 max latency.
> 5001 records sent, 1000.0 records/sec (3.91 MB/sec), 49.0 ms avg latency, 
> 503.0 max latency.
> 5001 records sent, 1000.2 records/sec (3.91 MB/sec), 37.3 ms avg latency, 
> 594.0 max latency.
> 5001 records sent, 1000.2 records/sec (3.91 MB/sec), 32.6 ms avg latency, 
> 501.0 max latency.
> 5000 records sent, 999.8 records/sec (3.91 MB/sec), 49.4 ms avg latency, 
> 516.0 max latency.
> org.apache.kafka.common.errors.NetworkException: The server disconnected 
> before a response was received.
> org.apache.kafka.common.errors.NetworkException: The server disconnected 
> before a response was received.
> org.apache.kafka.common.errors.NetworkException: The server disconnected 
> before a response was received.
> truncated
> 5001 records sent, 1000.2 records/sec (3.91 MB/sec), 33.9 ms avg latency, 
> 497.0 max latency.
> 4928 records sent, 985.6 records/sec (3.85 MB/sec), 42.1 ms avg latency, 
> 521.0 max latency.
> 5073 records sent, 1014.4 records/sec (3.96 MB/sec), 39.4 ms avg latency, 
> 418.0 max latency.
> 10 records sent, 999.950002 records/sec (3.91 MB/sec), 37.65 ms avg 
> latency, 798.00 ms max latency, 1 ms 50th, 260 ms 95th, 411 ms 99th, 571 ms 
> 99.9th.
> **consumer command:
> $ bin/kafka-console-consumer.sh --zookeeper 
> x.x.x.x:2181/dcos-service-kafka-framework --topic  test3R3P3  
> 1>~/kafka_output.log
> message stored:
> $ wc -l ~/kafka_output.log
> 99932 /home/montana/kafka_output.log
> I found only 99932 message are stored and 68 messages are lost.
> **topic describe command:
>  $ bin/kafka-topics.sh  --zookeeper x.x.x.x:2181/dcos-service-kafka-framework 
> --describe |grep test3R3
> Topic:test3R3P3 PartitionCount:3ReplicationFactor:3 Configs:
> Topic: test3R3P3Partition: 0Leader: 2   Replicas: 
> 1,2,0 Isr: 2,0,1
> Topic: test3R3P3Partition: 1Leader: 2   Replicas: 
> 2,0,1 Isr: 2,0,1
> Topic: test3R3P3Partition: 2Leader: 0   Replicas: 
> 0,1,2 Isr: 2,0,1
> **consumer group command:
> $  bin/kafka-consumer-groups.sh --zookeeper 
> x.x.x.x:2181/dcos-service-kafka-framework --describe --group 
> console-consumer-9926
> GROUP  TOPIC  PARTITION  
> CURRENT-OFFSET  LOG-END-OFFSET  LAG OWNER
> console-consumer-9926  test3R3P3  0  
> 33265   33265   0   
> console-consumer-9926_node-44a8422fe1a0-1484127474935-c795478e-0
> console-consumer-9926  test3R3P3  1  
> 4   4   0   
> console-consumer-9926_node-44a8422fe1a0-1484127474935-c795478e-0
> console-consumer-9926  test3R3P3  2  
> 3   3   0   
> console-consumer-9926_node-44a8422fe1a0-1484127474935-c795478e-0
> could you please help me understand what this error means "err - 
> org.apache.kafka.common.errors.NetworkException: The server disconnected 
> before a response was received."?
> Could you please provide suggestion to fix this issue?
> we are seeing this behavior every-time we perform above test-scenario.
> my understanding is, there should not any data loss till n-1 broker is alive. 
> is message loss is an expected behavior in the above case?
> thanks
> Sandeep



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4616) Message log is seen when kafka-producer-perf-test.sh is running and any broker restarted in middle in-between

[sandeep kumar singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821003#comment-15821003
 ] 

sandeep kumar singh commented on KAFKA-4616:


thanks for reply. i applied acks=-1 option, but still see message loss.

command i ran:
$ bin/kafka-producer-perf-test.sh --num-records 10 --record-size 4096 
--throughput 5000 --topic test2R3P3 --producer-props 
bootstrap.servers=localhost:9092,localhost:9093,localhost:9094 acks=-1
8890 records sent, 1777.3 records/sec (6.94 MB/sec), 2039.2 ms avg latency, 
3282.0 max latency.
12342 records sent, 2468.4 records/sec (9.64 MB/sec), 2648.8 ms avg latency, 
3448.0 max latency.
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is 
not the leader for that topic-partition.
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is 
not the leader for that topic-partition.
truncated
org.apache.kafka.common.errors.NetworkException: The server disconnected before 
a response was received.
...truncated
10 records sent, 3716.504999 records/sec (14.52 MB/sec), 1565.19 ms avg 
latency, 3634.00 ms max latency, 1470 ms 50th, 3205 ms 95th, 3357 ms 99th, 3502 
ms 99.9th.

$ bin/kafka-consumer-groups.sh --zookeeper 127.0.0.1:2181 --describe --group 
console-consumer-96681
GROUP  TOPIC  PARTITION  
CURRENT-OFFSET  LOG-END-OFFSET  LAG OWNER
console-consumer-96681 test2R3P3  0  3  
 3   0   
console-consumer-96681_localhost.localdomain-1482869188877-44ac0d84-0
console-consumer-96681 test2R3P3  1  33271  
 33271   0   
console-consumer-96681_localhost.localdomain-1482869188877-44ac0d84-0
console-consumer-96681 test2R3P3  2  3  
 3   0   
console-consumer-96681_localhost.localdomain-1482869188877-44ac0d84-0

i send 10 messages but could see only 99937 messages get stored.


> Message log is seen when kafka-producer-perf-test.sh is running and any 
> broker restarted in middle in-between 
> --
>
> Key: KAFKA-4616
> URL: https://issues.apache.org/jira/browse/KAFKA-4616
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
> Environment: Apache mesos
>Reporter: sandeep kumar singh
>
> if any broker is restarted while kafka-producer-perf-test.sh command is 
> running, we see message loss.
> commands i run:
> **perf command:
> $ bin/kafka-producer-perf-test.sh --num-records 10 --record-size 4096  
> --throughput 1000 --topic test3R3P3 --producer-props 
> bootstrap.servers=x.x.x.x:,x.x.x.x:,x.x.x.x:
> I am  sending 10 messages of each having size 4096
> error thrown by perf command:
> 4944 records sent, 988.6 records/sec (3.86 MB/sec), 31.5 ms avg latency, 
> 433.0 max latency.
> 5061 records sent, 1012.0 records/sec (3.95 MB/sec), 67.7 ms avg latency, 
> 798.0 max latency.
> 5001 records sent, 1000.0 records/sec (3.91 MB/sec), 49.0 ms avg latency, 
> 503.0 max latency.
> 5001 records sent, 1000.2 records/sec (3.91 MB/sec), 37.3 ms avg latency, 
> 594.0 max latency.
> 5001 records sent, 1000.2 records/sec (3.91 MB/sec), 32.6 ms avg latency, 
> 501.0 max latency.
> 5000 records sent, 999.8 records/sec (3.91 MB/sec), 49.4 ms avg latency, 
> 516.0 max latency.
> org.apache.kafka.common.errors.NetworkException: The server disconnected 
> before a response was received.
> org.apache.kafka.common.errors.NetworkException: The server disconnected 
> before a response was received.
> org.apache.kafka.common.errors.NetworkException: The server disconnected 
> before a response was received.
> truncated
> 5001 records sent, 1000.2 records/sec (3.91 MB/sec), 33.9 ms avg latency, 
> 497.0 max latency.
> 4928 records sent, 985.6 records/sec (3.85 MB/sec), 42.1 ms avg latency, 
> 521.0 max latency.
> 5073 records sent, 1014.4 records/sec (3.96 MB/sec), 39.4 ms avg latency, 
> 418.0 max latency.
> 10 records sent, 999.950002 records/sec (3.91 MB/sec), 37.65 ms avg 
> latency, 798.00 ms max latency, 1 ms 50th, 260 ms 95th, 411 ms 99th, 571 ms 
> 99.9th.
> **consumer command:
> $ bin/kafka-console-consumer.sh --zookeeper 
> x.x.x.x:2181/dcos-service-kafka-framework --topic  test3R3P3  
> 1>~/kafka_output.log
> message stored:
> $ wc -l ~/kafka_output.log
> 99932 /home/montana/kafka_output.log
> I found only 99932 message are stored and 68 messages are lost.
> **topic describe command:
>  $ bin/kafka-topics.sh  --zookeeper x.x.x.x:2181/dcos-service-kafka-framework 
> --describe |grep test3R3
> Topic:test3R3P3 PartitionCount:3

[jira] [Updated] (KAFKA-4614) Long GC pause harming broker performance which is caused by mmap objects created for OffsetIndex

[Yuto Kawamura (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuto Kawamura updated KAFKA-4614:
-
Status: Patch Available  (was: Open)

> Long GC pause harming broker performance which is caused by mmap objects 
> created for OffsetIndex
> 
>
> Key: KAFKA-4614
> URL: https://issues.apache.org/jira/browse/KAFKA-4614
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.1
>Reporter: Yuto Kawamura
>Assignee: Yuto Kawamura
>  Labels: latency, performance
>
> First let me clarify our system environment information as I think it's 
> important to understand this issue:
> OS: CentOS6
> Kernel version: 2.6.32-XX
> Filesystem used for data volume: XFS
> Java version: 1.8.0_66
> GC option: Kafka default(G1GC)
> Kafka version: 0.10.0.1
> h2. Phenomenon
> In our Kafka cluster, an usual response time for Produce request is about 1ms 
> for 50th percentile to 10ms for 99th percentile. All topics are configured to 
> have 3 replicas and all producers are configured {{acks=all}} so this time 
> includes replication latency.
> Sometimes we observe 99th percentile latency get increased to 100ms ~ 500ms 
> but for the most cases the time consuming part is "Remote" which means it is 
> caused by slow replication which is known to happen by various reasons(which 
> is also an issue that we're trying to improve, but out of interest within 
> this issue).
> However, we found that there are some different patterns which happens rarely 
> but stationary 3 ~ 5 times a day for each servers. The interesting part is 
> that "RequestQueue" also got increased as well as "Total" and "Remote".
> At the same time, we observed that the disk read metrics(in terms of both 
> read bytes and read time) spikes exactly for the same moment. Currently we 
> have only caught up consumers so this metric sticks to zero while all 
> requests are served by page cache.
> In order to investigate what Kafka is "read"ing, I employed SystemTap and 
> wrote the following script. It traces all disk reads(only actual read by 
> physical device) made for the data volume by broker process.
> {code}
> global target_pid = KAFKA_PID
> global target_dev = DATA_VOLUME
> probe ioblock.request {
>   if (rw == BIO_READ && pid() == target_pid && devname == target_dev) {
>  t_ms = gettimeofday_ms() + 9 * 3600 * 1000 // timezone adjustment
>  printf("%s,%03d:  tid = %d, device = %s, inode = %d, size = %d\n", 
> ctime(t_ms / 1000), t_ms % 1000, tid(), devname, ino, size)
>  print_backtrace()
>  print_ubacktrace()
>   }
> }
> {code}
> As the result, we could observe many logs like below:
> {code}
> Thu Dec 22 17:21:39 2016,209:  tid = 126123, device = sdb1, inode = -1, size 
> = 4096
>  0x81275050 : generic_make_request+0x0/0x5a0 [kernel]
>  0x81275660 : submit_bio+0x70/0x120 [kernel]
>  0xa036bcaa : _xfs_buf_ioapply+0x16a/0x200 [xfs]
>  0xa036d95f : xfs_buf_iorequest+0x4f/0xe0 [xfs]
>  0xa036db46 : _xfs_buf_read+0x36/0x60 [xfs]
>  0xa036dc1b : xfs_buf_read+0xab/0x100 [xfs]
>  0xa0363477 : xfs_trans_read_buf+0x1f7/0x410 [xfs]
>  0xa033014e : xfs_btree_read_buf_block+0x5e/0xd0 [xfs]
>  0xa0330854 : xfs_btree_lookup_get_block+0x84/0xf0 [xfs]
>  0xa0330edf : xfs_btree_lookup+0xbf/0x470 [xfs]
>  0xa032456f : xfs_bmbt_lookup_eq+0x1f/0x30 [xfs]
>  0xa032628b : xfs_bmap_del_extent+0x12b/0xac0 [xfs]
>  0xa0326f34 : xfs_bunmapi+0x314/0x850 [xfs]
>  0xa034ad79 : xfs_itruncate_extents+0xe9/0x280 [xfs]
>  0xa0366de5 : xfs_inactive+0x2f5/0x450 [xfs]
>  0xa0374620 : xfs_fs_clear_inode+0xa0/0xd0 [xfs]
>  0x811affbc : clear_inode+0xac/0x140 [kernel]
>  0x811b0776 : generic_delete_inode+0x196/0x1d0 [kernel]
>  0x811b0815 : generic_drop_inode+0x65/0x80 [kernel]
>  0x811af662 : iput+0x62/0x70 [kernel]
>  0x37ff2e5347 : munmap+0x7/0x30 [/lib64/libc-2.12.so]
>  0x7ff169ba5d47 : Java_sun_nio_ch_FileChannelImpl_unmap0+0x17/0x50 
> [/usr/jdk1.8.0_66/jre/lib/amd64/libnio.so]
>  0x7ff269a1307e
> {code}
> We took a jstack dump of the broker process and found that tid = 126123 
> corresponds to the thread which is for GC(hex(tid) == nid(Native Thread ID)):
> {code}
> $ grep 0x1ecab /tmp/jstack.dump
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x7ff278d0c800 
> nid=0x1ecab in Object.wait() [0x7ff17da11000]
> {code}
> In order to confirm, we enabled {{PrintGCApplicationStoppedTime}} switch and 
> confirmed that in total the time the broker paused is longer than usual, when 
> a broker experiencing the issue.
> {code}
> $ grep --text 'Total time for which application threads were stopped' 

[jira] [Commented] (KAFKA-4614) Long GC pause harming broker performance which is caused by mmap objects created for OffsetIndex

[Yuto Kawamura (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820948#comment-15820948
 ] 

Yuto Kawamura commented on KAFKA-4614:
--

I applied the patch to one of our production brokers, deployed and observe over 
24 hours.
As the result, this issue disappeared completely. As I wrote in description, it 
was happening at least 3 ~ 4 times a day so I think we can say the patch solved 
the problem.

[~ijuma] Filled up a PR. Please review if possible.

[~huxi_2b]
Thanks for your comments.

> Kafka already offers a method named 'forceUnmap' that actually does this kind 
> of thing
this is exactly what I did ;)

> If others use MappedByteBuffer later, it will crash.
Right. I think it is somewhat a dangerous solution, but still think its safe 
because access to mmap object is limited to {{AbstractIndex}} and it's 
subclasses. Indeed we need to be careful for changing things around there 
though.

> Portability risk, although sun.misc.Cleaner is present in most of major JVM 
> vendors.
Right, but this class is already imported to support windows so I don't think 
this is a problem.

> System.gc()
I think this doesn't work at all. AFAIK:
- System.gc() does not guarantee(JVM does "best effort") to collect all garbage 
which are available to be collected.
- No matter if GC triggered manually or not, it blocks application threads by 
the same reason.

[~miguno] Thanks. The investigation was hard but was a great opportunity to 
learn many new things ;)


> Long GC pause harming broker performance which is caused by mmap objects 
> created for OffsetIndex
> 
>
> Key: KAFKA-4614
> URL: https://issues.apache.org/jira/browse/KAFKA-4614
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.1
>Reporter: Yuto Kawamura
>Assignee: Yuto Kawamura
>  Labels: latency, performance
>
> First let me clarify our system environment information as I think it's 
> important to understand this issue:
> OS: CentOS6
> Kernel version: 2.6.32-XX
> Filesystem used for data volume: XFS
> Java version: 1.8.0_66
> GC option: Kafka default(G1GC)
> Kafka version: 0.10.0.1
> h2. Phenomenon
> In our Kafka cluster, an usual response time for Produce request is about 1ms 
> for 50th percentile to 10ms for 99th percentile. All topics are configured to 
> have 3 replicas and all producers are configured {{acks=all}} so this time 
> includes replication latency.
> Sometimes we observe 99th percentile latency get increased to 100ms ~ 500ms 
> but for the most cases the time consuming part is "Remote" which means it is 
> caused by slow replication which is known to happen by various reasons(which 
> is also an issue that we're trying to improve, but out of interest within 
> this issue).
> However, we found that there are some different patterns which happens rarely 
> but stationary 3 ~ 5 times a day for each servers. The interesting part is 
> that "RequestQueue" also got increased as well as "Total" and "Remote".
> At the same time, we observed that the disk read metrics(in terms of both 
> read bytes and read time) spikes exactly for the same moment. Currently we 
> have only caught up consumers so this metric sticks to zero while all 
> requests are served by page cache.
> In order to investigate what Kafka is "read"ing, I employed SystemTap and 
> wrote the following script. It traces all disk reads(only actual read by 
> physical device) made for the data volume by broker process.
> {code}
> global target_pid = KAFKA_PID
> global target_dev = DATA_VOLUME
> probe ioblock.request {
>   if (rw == BIO_READ && pid() == target_pid && devname == target_dev) {
>  t_ms = gettimeofday_ms() + 9 * 3600 * 1000 // timezone adjustment
>  printf("%s,%03d:  tid = %d, device = %s, inode = %d, size = %d\n", 
> ctime(t_ms / 1000), t_ms % 1000, tid(), devname, ino, size)
>  print_backtrace()
>  print_ubacktrace()
>   }
> }
> {code}
> As the result, we could observe many logs like below:
> {code}
> Thu Dec 22 17:21:39 2016,209:  tid = 126123, device = sdb1, inode = -1, size 
> = 4096
>  0x81275050 : generic_make_request+0x0/0x5a0 [kernel]
>  0x81275660 : submit_bio+0x70/0x120 [kernel]
>  0xa036bcaa : _xfs_buf_ioapply+0x16a/0x200 [xfs]
>  0xa036d95f : xfs_buf_iorequest+0x4f/0xe0 [xfs]
>  0xa036db46 : _xfs_buf_read+0x36/0x60 [xfs]
>  0xa036dc1b : xfs_buf_read+0xab/0x100 [xfs]
>  0xa0363477 : xfs_trans_read_buf+0x1f7/0x410 [xfs]
>  0xa033014e : xfs_btree_read_buf_block+0x5e/0xd0 [xfs]
>  0xa0330854 : xfs_btree_lookup_get_block+0x84/0xf0 [xfs]
>  0xa0330edf : xfs_btree_lookup+0xbf/0x470 [xfs]
>  0xa032456f : xfs_bmbt_lookup_eq+0x1f/0x30 [xfs]

[jira] [Commented] (KAFKA-4610) getting error:Batch containing 3 record(s) expired due to timeout while requesting metadata from brokers for test2R2P2-1

[sandeep kumar singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820941#comment-15820941
 ] 

sandeep kumar singh commented on KAFKA-4610:


thanks for the update. i am trying to check in broker logs. but non of the 
brokers throwing any exceptions when this error occurred. i am seeing this 
error "ERROR Error when sending message to topic test3R3P3 with key: null, 
value: 4096 bytes with error: 
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Batch containing 3 record(s) 
expired due to timeout while requesting metadata from brokers for test3R3P3-0" 
every time i run producer.  

producer command - cat kafka_output.log | bin/kafka-console-producer.sh 
--broker-list localhost:9092,localhost:9093,localhost:9094 --batch-size 1000 
--message-send-max-retries 10 --request-required-acks -1 --topic test3R3P3

kafka_output.log has 10 records of 4096 length each

i see this error even when all brokers are healthy and all partitions have 
valid leaders. 

are you saying brokers restarts internally may be due to load? but when i check 
the UNIX Process ID of brokers before and after running broker, i see the PID 
is same. which means the brokers are not restarted.

> getting error:Batch containing 3 record(s) expired due to timeout while 
> requesting metadata from brokers for test2R2P2-1
> 
>
> Key: KAFKA-4610
> URL: https://issues.apache.org/jira/browse/KAFKA-4610
> Project: Kafka
>  Issue Type: Bug
> Environment: Dev
>Reporter: sandeep kumar singh
>
> i a getting below error when running producer client, which take messages 
> from an input file kafka_message.log. this log file is pilled with 10 
> records per second of each message of length 4096
> error - 
> [2017-01-09 14:45:24,813] ERROR Error when sending message to topic test2R2P2 
> with key: null, value: 4096 bytes with error: 
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Batch containing 3 record(s) 
> expired due to timeout while requesting metadata from brokers for test2R2P2-0
> [2017-01-09 14:45:24,816] ERROR Error when sending message to topic test2R2P2 
> with key: null, value: 4096 bytes with error: 
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Batch containing 3 record(s) 
> expired due to timeout while requesting metadata from brokers for test2R2P2-0
> [2017-01-09 14:45:24,816] ERROR Error when sending message to topic test2R2P2 
> with key: null, value: 4096 bytes with error: 
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Batch containing 3 record(s) 
> expired due to timeout while requesting metadata from brokers for test2R2P2-0
> [2017-01-09 14:45:24,816] ERROR Error when sending message to topic test2R2P2 
> with key: null, value: 4096 bytes with error: 
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Batch containing 3 record(s) 
> expired due to timeout while requesting metadata from brokers for test2R2P2-0
> [2017-01-09 14:45:24,816] ERROR Error when sending message to topic test2R2P2 
> with key: null, value: 4096 bytes with error: 
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Batch containing 3 record(s) 
> expired due to timeout while requesting metadata from brokers for test2R2P2-0
> command i run :
> $ bin/kafka-console-producer.sh --broker-list x.x.x.x:,x.x.x.x: 
> --batch-size 1000 --message-send-max-retries 10 --request-required-acks 1 
> --topic test2R2P2 <~/kafka_message.log
> there are 2 brokers running and the topic has partitions = 2 and replication 
> factor 2. 
> Could you please help me understand what does that error means?
> also i see message loss when i manually restart one of the broker and while 
> kafak-producer-perf-test command is running? is this a expected behavior?
> thanks
> Sandeep



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2353: MINOR: Remove unnecessary options in SCRAM test ja...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/2353

MINOR: Remove unnecessary options in SCRAM test jaas config



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka minor-scram

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2353.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2353


commit d89a156503f9dd1cf0d4d9b1f66d5320f26956b6
Author: Rajini Sivaram 
Date:   2017-01-12T12:38:30Z

MINOR: Remove unnecessary options in SCRAM test jaas config




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2352: KAFKA-4614 Forcefully unmap mmap of OffsetIndex to...

[kawamuray]

GitHub user kawamuray opened a pull request:

https://github.com/apache/kafka/pull/2352

KAFKA-4614 Forcefully unmap mmap of OffsetIndex to prevent long GC pause

Issue: https://issues.apache.org/jira/browse/KAFKA-4614

Fixes the problem that the broker threads suffered by long GC pause.
When GC thread collects mmap objects which were created for index files, it 
unmaps memory mapping so kernel turns to delete a file physically. This work 
may transparently read file's metadata from physical disk if it's not available 
on cache.
This seems to happen typically when we're using G1GC, due to it's strategy 
to left a garbage for a long time if other objects in the same region are still 
alive.
See the link for the details.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kawamuray/kafka 
KAFKA-4614-force-munmap-for-index

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2352.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2352


commit 8c3a4c9f5a6188c641a45cb3e111094f545f7e66
Author: Yuto Kawamura 
Date:   2017-01-11T08:10:05Z

KAFKA-4614 Forcefully unmap mmap of OffsetIndex to prevent long GC pause




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4614) Long GC pause harming broker performance which is caused by mmap objects created for OffsetIndex

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820924#comment-15820924
 ] 

ASF GitHub Bot commented on KAFKA-4614:
---

GitHub user kawamuray opened a pull request:

https://github.com/apache/kafka/pull/2352

KAFKA-4614 Forcefully unmap mmap of OffsetIndex to prevent long GC pause

Issue: https://issues.apache.org/jira/browse/KAFKA-4614

Fixes the problem that the broker threads suffered by long GC pause.
When GC thread collects mmap objects which were created for index files, it 
unmaps memory mapping so kernel turns to delete a file physically. This work 
may transparently read file's metadata from physical disk if it's not available 
on cache.
This seems to happen typically when we're using G1GC, due to it's strategy 
to left a garbage for a long time if other objects in the same region are still 
alive.
See the link for the details.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kawamuray/kafka 
KAFKA-4614-force-munmap-for-index

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2352.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2352


commit 8c3a4c9f5a6188c641a45cb3e111094f545f7e66
Author: Yuto Kawamura 
Date:   2017-01-11T08:10:05Z

KAFKA-4614 Forcefully unmap mmap of OffsetIndex to prevent long GC pause




> Long GC pause harming broker performance which is caused by mmap objects 
> created for OffsetIndex
> 
>
> Key: KAFKA-4614
> URL: https://issues.apache.org/jira/browse/KAFKA-4614
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.1
>Reporter: Yuto Kawamura
>Assignee: Yuto Kawamura
>  Labels: latency, performance
>
> First let me clarify our system environment information as I think it's 
> important to understand this issue:
> OS: CentOS6
> Kernel version: 2.6.32-XX
> Filesystem used for data volume: XFS
> Java version: 1.8.0_66
> GC option: Kafka default(G1GC)
> Kafka version: 0.10.0.1
> h2. Phenomenon
> In our Kafka cluster, an usual response time for Produce request is about 1ms 
> for 50th percentile to 10ms for 99th percentile. All topics are configured to 
> have 3 replicas and all producers are configured {{acks=all}} so this time 
> includes replication latency.
> Sometimes we observe 99th percentile latency get increased to 100ms ~ 500ms 
> but for the most cases the time consuming part is "Remote" which means it is 
> caused by slow replication which is known to happen by various reasons(which 
> is also an issue that we're trying to improve, but out of interest within 
> this issue).
> However, we found that there are some different patterns which happens rarely 
> but stationary 3 ~ 5 times a day for each servers. The interesting part is 
> that "RequestQueue" also got increased as well as "Total" and "Remote".
> At the same time, we observed that the disk read metrics(in terms of both 
> read bytes and read time) spikes exactly for the same moment. Currently we 
> have only caught up consumers so this metric sticks to zero while all 
> requests are served by page cache.
> In order to investigate what Kafka is "read"ing, I employed SystemTap and 
> wrote the following script. It traces all disk reads(only actual read by 
> physical device) made for the data volume by broker process.
> {code}
> global target_pid = KAFKA_PID
> global target_dev = DATA_VOLUME
> probe ioblock.request {
>   if (rw == BIO_READ && pid() == target_pid && devname == target_dev) {
>  t_ms = gettimeofday_ms() + 9 * 3600 * 1000 // timezone adjustment
>  printf("%s,%03d:  tid = %d, device = %s, inode = %d, size = %d\n", 
> ctime(t_ms / 1000), t_ms % 1000, tid(), devname, ino, size)
>  print_backtrace()
>  print_ubacktrace()
>   }
> }
> {code}
> As the result, we could observe many logs like below:
> {code}
> Thu Dec 22 17:21:39 2016,209:  tid = 126123, device = sdb1, inode = -1, size 
> = 4096
>  0x81275050 : generic_make_request+0x0/0x5a0 [kernel]
>  0x81275660 : submit_bio+0x70/0x120 [kernel]
>  0xa036bcaa : _xfs_buf_ioapply+0x16a/0x200 [xfs]
>  0xa036d95f : xfs_buf_iorequest+0x4f/0xe0 [xfs]
>  0xa036db46 : _xfs_buf_read+0x36/0x60 [xfs]
>  0xa036dc1b : xfs_buf_read+0xab/0x100 [xfs]
>  0xa0363477 : xfs_trans_read_buf+0x1f7/0x410 [xfs]
>  0xa033014e : xfs_btree_read_buf_block+0x5e/0xd0 [xfs]
>  0xa0330854 : xfs_btree_lookup_get_block+0x84/0xf0 [xfs]
>  0xa0330edf : 

[jira] [Commented] (KAFKA-3992) InstanceAlreadyExistsException Error for Consumers Starting in Parallel

[darmagan (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820918#comment-15820918
 ] 

darmagan commented on KAFKA-3992:
-

I have the same problem. I have started 20 consumer_threads and it looks like 
they have all the same id. Same problem with client_id set.
How can I set the ids of the consumer threads?

Konfig:
input {
 kafka {
  topics => 'logstash_logs02'
  client_id => "oneapp.app.log.msp"
  group_id => "group02"
  consumer_threads => 20
 }
}

Error:
[2017-01-12T13:30:30,791][INFO ][org.apache.kafka.common.utils.AppInfoParser] 
Kafka version : 0.10.0.1
[2017-01-12T13:30:30,791][INFO ][org.apache.kafka.common.utils.AppInfoParser] 
Kafka commitId : a7a17cdec9eaa6c5
[2017-01-12T13:30:30,791][WARN ][org.apache.kafka.common.utils.AppInfoParser] 
Error registering AppInfo mbean
javax.management.InstanceAlreadyExistsException: 
kafka.consumer:type=app-info,id=oneapp.app.log.msp
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) 
~[?:1.8.0_111]
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
 ~[?:1.8.0_111]
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
 ~[?:1.8.0_111]
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
 ~[?:1.8.0_111]
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
 ~[?:1.8.0_111]
at 
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) 
~[?:1.8.0_111]


Regards

> InstanceAlreadyExistsException Error for Consumers Starting in Parallel
> ---
>
> Key: KAFKA-3992
> URL: https://issues.apache.org/jira/browse/KAFKA-3992
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Alexander Cook
>
> I see the following error sometimes when I start multiple consumers at about 
> the same time in the same process (separate threads). Everything seems to 
> work fine afterwards, so should this not actually be an ERROR level message, 
> or could there be something going wrong that I don't see? 
> Let me know if I can provide any more info! 
> Error processing messages: Error registering mbean 
> kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id=node--1
> org.apache.kafka.common.KafkaException: Error registering mbean 
> kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id=node--1
>  
> Caused by: javax.management.InstanceAlreadyExistsException: 
> kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id=node--1
> Here is the full stack trace: 
> M[?:com.ibm.streamsx.messaging.kafka.KafkaConsumerV9.produceTuples:-1]  - 
> Error processing messages: Error registering mbean 
> kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id=node--1
> org.apache.kafka.common.KafkaException: Error registering mbean 
> kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id=node--1
>   at 
> org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:159)
>   at 
> org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:77)
>   at 
> org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:288)
>   at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:177)
>   at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:162)
>   at 
> org.apache.kafka.common.network.Selector$SelectorMetrics.maybeRegisterConnectionMetrics(Selector.java:641)
>   at org.apache.kafka.common.network.Selector.poll(Selector.java:268)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:303)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:197)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:187)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:126)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorKnown(AbstractCoordinator.java:186)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:857)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:829)
>   at 
> com.ibm.streamsx.messaging.kafka.KafkaConsumerV9.produceTuples(KafkaConsumerV9.java:129)
>   at 
> 

[jira] [Commented] (KAFKA-1843) Metadata fetch/refresh in new producer should handle all node connection states gracefully

[sandeep kumar singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820892#comment-15820892
 ] 

sandeep kumar singh commented on KAFKA-1843:


in step 4 when you bring K1 up, should it not update/refresh metadata as 
(K1,K2) immediately, without worrying for  metadata.max.age.ms.

> Metadata fetch/refresh in new producer should handle all node connection 
> states gracefully
> --
>
> Key: KAFKA-1843
> URL: https://issues.apache.org/jira/browse/KAFKA-1843
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, producer 
>Affects Versions: 0.8.2.0
>Reporter: Ewen Cheslack-Postava
>  Labels: patch
> Fix For: 0.8.2.1
>
>
> KAFKA-1642 resolved some issues with the handling of broker connection states 
> to avoid high CPU usage, but made the minimal fix rather than the ideal one. 
> The code for handling the metadata fetch is difficult to get right because it 
> has to handle a lot of possible connectivity states and failure modes across 
> all the known nodes. It also needs to correctly integrate with the 
> surrounding event loop, providing correct poll() timeouts to both avoid busy 
> looping and make sure it wakes up and tries new nodes in the face of both 
> connection and request failures.
> A patch here should address a few issues:
> 1. Make sure connection timeouts, as implemented in KAFKA-1842, are cleanly 
> integrated. This mostly means that when a connecting node is selected to 
> fetch metadata from, that the code notices that and sets the next timeout 
> based on the connection timeout rather than some other backoff.
> 2. Rethink the logic and naming of NetworkClient.leastLoadedNode. That method 
> actually takes into account a) the current connectivity of each node, b) 
> whether the node had a recent connection failure, c) the "load" in terms of 
> in flight requests. It also needs to ensure that different clients don't use 
> the same ordering across multiple calls (which is already addressed in the 
> current code by nodeIndexOffset) and that we always eventually try all nodes 
> in the face of connection failures (which isn't currently handled by 
> leastLoadedNode and probably cannot be without tracking additional state). 
> This method also has to work for new consumer use cases even though it is 
> currently only used by the new producer's metadata fetch. Finally it has to 
> properly handle when other code calls initiateConnect() since the normal path 
> for sending messages also initiates connections.
> We can already say that there is an order of preference given a single call 
> (as follows), but making this work across multiple calls when some initial 
> choices fail to connect or return metadata *and* connection states may be 
> changing is much more difficult.
>  * Connected, zero in flight requests - the request can be sent immediately
>  * Connecting node - it will hopefully be connected very soon and by 
> definition has no in flight requests
>  * Disconnected - same reasoning as for a connecting node
>  * Connected, > 0 in flight requests - we consider any # of in flight 
> requests as a big enough backlog to delay the request a lot.
> We could use an approach that better accounts for # of in flight requests 
> rather than just turning it into a boolean variable, but that probably 
> introduces much more complexity than it is worth.
> 3. The most difficult case to handle so far has been when leastLoadedNode 
> returns a disconnected node to maybeUpdateMetadata as its best option. 
> Properly handling the two resulting cases (initiateConnect fails immediately 
> vs. taking some time to possibly establish the connection) is tricky.
> 4. Consider optimizing for the failure cases. The most common cases are when 
> you already have an active connection and can immediately get the metadata or 
> you need to establish a connection, but the connection and metadata 
> request/response happen very quickly. These common cases are infrequent 
> enough (default every 5 min) that establishing an extra connection isn't a 
> big deal as long as it's eventually cleaned up. The edge cases, like network 
> partitions where some subset of nodes become unreachable for a long period, 
> are harder to reason about but we should be sure we will always be able to 
> gracefully recover from them.
> KAFKA-1642 enumerated the possible outcomes of a single call to 
> maybeUpdateMetadata. A good fix for this would consider all of those outcomes 
> for repeated calls to 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4614) Long GC pause harming broker performance which is caused by mmap objects created for OffsetIndex

[Michael Noll (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820875#comment-15820875
 ] 

Michael Noll commented on KAFKA-4614:
-

[~kawamuray] Sorry to hear that you ran into this.  Thanks a lot for the 
thorough investigation!

> Long GC pause harming broker performance which is caused by mmap objects 
> created for OffsetIndex
> 
>
> Key: KAFKA-4614
> URL: https://issues.apache.org/jira/browse/KAFKA-4614
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.1
>Reporter: Yuto Kawamura
>Assignee: Yuto Kawamura
>  Labels: latency, performance
>
> First let me clarify our system environment information as I think it's 
> important to understand this issue:
> OS: CentOS6
> Kernel version: 2.6.32-XX
> Filesystem used for data volume: XFS
> Java version: 1.8.0_66
> GC option: Kafka default(G1GC)
> Kafka version: 0.10.0.1
> h2. Phenomenon
> In our Kafka cluster, an usual response time for Produce request is about 1ms 
> for 50th percentile to 10ms for 99th percentile. All topics are configured to 
> have 3 replicas and all producers are configured {{acks=all}} so this time 
> includes replication latency.
> Sometimes we observe 99th percentile latency get increased to 100ms ~ 500ms 
> but for the most cases the time consuming part is "Remote" which means it is 
> caused by slow replication which is known to happen by various reasons(which 
> is also an issue that we're trying to improve, but out of interest within 
> this issue).
> However, we found that there are some different patterns which happens rarely 
> but stationary 3 ~ 5 times a day for each servers. The interesting part is 
> that "RequestQueue" also got increased as well as "Total" and "Remote".
> At the same time, we observed that the disk read metrics(in terms of both 
> read bytes and read time) spikes exactly for the same moment. Currently we 
> have only caught up consumers so this metric sticks to zero while all 
> requests are served by page cache.
> In order to investigate what Kafka is "read"ing, I employed SystemTap and 
> wrote the following script. It traces all disk reads(only actual read by 
> physical device) made for the data volume by broker process.
> {code}
> global target_pid = KAFKA_PID
> global target_dev = DATA_VOLUME
> probe ioblock.request {
>   if (rw == BIO_READ && pid() == target_pid && devname == target_dev) {
>  t_ms = gettimeofday_ms() + 9 * 3600 * 1000 // timezone adjustment
>  printf("%s,%03d:  tid = %d, device = %s, inode = %d, size = %d\n", 
> ctime(t_ms / 1000), t_ms % 1000, tid(), devname, ino, size)
>  print_backtrace()
>  print_ubacktrace()
>   }
> }
> {code}
> As the result, we could observe many logs like below:
> {code}
> Thu Dec 22 17:21:39 2016,209:  tid = 126123, device = sdb1, inode = -1, size 
> = 4096
>  0x81275050 : generic_make_request+0x0/0x5a0 [kernel]
>  0x81275660 : submit_bio+0x70/0x120 [kernel]
>  0xa036bcaa : _xfs_buf_ioapply+0x16a/0x200 [xfs]
>  0xa036d95f : xfs_buf_iorequest+0x4f/0xe0 [xfs]
>  0xa036db46 : _xfs_buf_read+0x36/0x60 [xfs]
>  0xa036dc1b : xfs_buf_read+0xab/0x100 [xfs]
>  0xa0363477 : xfs_trans_read_buf+0x1f7/0x410 [xfs]
>  0xa033014e : xfs_btree_read_buf_block+0x5e/0xd0 [xfs]
>  0xa0330854 : xfs_btree_lookup_get_block+0x84/0xf0 [xfs]
>  0xa0330edf : xfs_btree_lookup+0xbf/0x470 [xfs]
>  0xa032456f : xfs_bmbt_lookup_eq+0x1f/0x30 [xfs]
>  0xa032628b : xfs_bmap_del_extent+0x12b/0xac0 [xfs]
>  0xa0326f34 : xfs_bunmapi+0x314/0x850 [xfs]
>  0xa034ad79 : xfs_itruncate_extents+0xe9/0x280 [xfs]
>  0xa0366de5 : xfs_inactive+0x2f5/0x450 [xfs]
>  0xa0374620 : xfs_fs_clear_inode+0xa0/0xd0 [xfs]
>  0x811affbc : clear_inode+0xac/0x140 [kernel]
>  0x811b0776 : generic_delete_inode+0x196/0x1d0 [kernel]
>  0x811b0815 : generic_drop_inode+0x65/0x80 [kernel]
>  0x811af662 : iput+0x62/0x70 [kernel]
>  0x37ff2e5347 : munmap+0x7/0x30 [/lib64/libc-2.12.so]
>  0x7ff169ba5d47 : Java_sun_nio_ch_FileChannelImpl_unmap0+0x17/0x50 
> [/usr/jdk1.8.0_66/jre/lib/amd64/libnio.so]
>  0x7ff269a1307e
> {code}
> We took a jstack dump of the broker process and found that tid = 126123 
> corresponds to the thread which is for GC(hex(tid) == nid(Native Thread ID)):
> {code}
> $ grep 0x1ecab /tmp/jstack.dump
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x7ff278d0c800 
> nid=0x1ecab in Object.wait() [0x7ff17da11000]
> {code}
> In order to confirm, we enabled {{PrintGCApplicationStoppedTime}} switch and 
> confirmed that in total the time the broker paused is longer than usual, when 
> a broker 

FOSDEM 2017 Open Source Conference - Brussels

[Sharan F]

Hello Everyone

This email is to tell you about ASF participation at FOSDEM. The event 
will be held in Brussels on 4^th & 5^th February 2017 and we are hoping 
that many people from our ASF projects will be there.


https://fosdem.org/2017/

Attending FOSDEM is completely free and the ASF will again be running a 
booth there. Our main focus will on talking to people about the ASF, our 
projects and communities.


*_Why Attend FOSDEM?_*
Some reasons for attending FOSDEM are:

1. Promoting your project: FOSDEM has up to 4-5000 attendees so is a
   great place to spread the word about your project
2. Learning, participating and meeting up: FOSDEM is a developers
   conference so includes presentations covering a range of
   technologies and includes lots of topic specific devrooms

_*FOSDEM Wiki *_
A page on the Community Development wiki has been created with the main 
details about our involvement at conference, so please take a look


https://cwiki.apache.org/confluence/display/COMDEV/FOSDEM+2017

If you would like to spend some time on the ASF booth promoting your 
project then please sign up on the FOSDEM wiki page. Initially we would 
like to split this into slots of 3-4 hours but this will depend on the 
number of projects that are represented.


We are also looking for volunteers to help out on the booth over the 2 
days of the conference, so if you are going to be there and are willing 
to help then please add your name to the volunteer list.


_*Project Stickers*_
If you are going to be at FOSDEM and do not have any project stickers to 
give away then we may (budget permitting) be able to help you get some 
printed. Please contact me with your requirements.


_*Social Event*_
Some people have asked about organising an ASF social event / meetup 
during the conference. This is possible but we will need know how many 
people are interested and which date works best. The FOSDEM wiki page 
also contains an 'Arrival / Departure' section so so please add your 
details if you would like to participate.


I hope this helps people see some of the advantages of attending FOSDEM 
and we are looking forward to seeing lots of people there from our ASF 
communities.


Thanks
Sharan

Apache Community Development
http://community.apache.org/

[GitHub] kafka pull request #2351: MINOR: add Eclipse generated files to toplevel .gi...

[edoardocomar]

GitHub user edoardocomar opened a pull request:

https://github.com/apache/kafka/pull/2351

MINOR: add Eclipse generated files to toplevel .gitignore

Ignoring sibfolders' .gitignore except tests
Ignoring core/cache-main and core/cache-tests

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/edoardocomar/kafka minor-gitignore

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2351.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2351


commit 8cbd786a72e9cee2facb8a59ee203c0dbb8b8979
Author: Edoardo Comar 
Date:   2017-01-12T10:50:56Z

MINOR: add Eclipse generated files to toplevel .gitignore 

Ignoring sibfolders' .gitignore except tests
Ignoring core/cache-main and core/cache-tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2350: MINOR: Finished exposing the broker config

[enothereska]

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/2350

MINOR: Finished exposing the broker config

This was left over from KIP-104. Thanks to @ijuma for pointing out.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka minor-broker-level-config

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2350.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2350


commit 70e48d7ff17f0f536e96406148f83421b3ae4ea3
Author: Eno Thereska 
Date:   2017-01-12T10:39:11Z

Finished exposing the broker config




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2349: KAFKA-4603 Command parsed error, change all Option...

[auroraxlh]

GitHub user auroraxlh opened a pull request:

https://github.com/apache/kafka/pull/2349

KAFKA-4603 Command parsed error, change all OptionParser constructor 

KAFKA-4603 the command parsed error
Using "new OptionParser" might result in parse error

Change all the OptionParser constructor in Kafka into "new 
OptionParser(false)"

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/auroraxlh/kafka fix_OptionParser_bug

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2349.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2349


commit 749fa3786f1b283409462698856369f20793ef57
Author: auroraxlh 
Date:   2017-01-12T09:35:23Z

KAFKA-4603 the command parsed error

Using "new OptionParser" might result in parse error

Change all the OptionParser constructor in Kafka into "new OptionParser(false)"




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4603) the argument of shell in doc wrong and command parsed error

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820676#comment-15820676
 ] 

ASF GitHub Bot commented on KAFKA-4603:
---

GitHub user auroraxlh opened a pull request:

https://github.com/apache/kafka/pull/2349

KAFKA-4603 Command parsed error, change all OptionParser constructor 

KAFKA-4603 the command parsed error
Using "new OptionParser" might result in parse error

Change all the OptionParser constructor in Kafka into "new 
OptionParser(false)"

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/auroraxlh/kafka fix_OptionParser_bug

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2349.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2349


commit 749fa3786f1b283409462698856369f20793ef57
Author: auroraxlh 
Date:   2017-01-12T09:35:23Z

KAFKA-4603 the command parsed error

Using "new OptionParser" might result in parse error

Change all the OptionParser constructor in Kafka into "new OptionParser(false)"




> the argument of shell in doc wrong and command parsed error
> ---
>
> Key: KAFKA-4603
> URL: https://issues.apache.org/jira/browse/KAFKA-4603
> Project: Kafka
>  Issue Type: Bug
>  Components: admin, documentation
>Affects Versions: 0.10.0.1, 0.10.2.0
> Environment: suse
>Reporter: Xin
>Assignee: Xin
>Priority: Minor
>
> according to the 7.6.2 Migrating clusters of document :
> ./zookeeper-security-migration.sh --zookeeper.acl=secure 
> --zookeeper.connection=localhost:2181
> joptsimple.OptionArgumentConversionException: Cannot parse argument 
> 'localhost:2181' of option zookeeper.connection.timeout
>   at joptsimple.AbstractOptionSpec.convertWith(AbstractOptionSpec.java:93)
>   at 
> joptsimple.ArgumentAcceptingOptionSpec.convert(ArgumentAcceptingOptionSpec.java:274)
>   at joptsimple.OptionSet.valuesOf(OptionSet.java:223)
>   at joptsimple.OptionSet.valueOf(OptionSet.java:172)
>   at kafka.admin.ZkSecurityMigrator$.run(ZkSecurityMigrator.scala:111)
>   at kafka.admin.ZkSecurityMigrator$.main(ZkSecurityMigrator.scala:119)
>   at kafka.admin.ZkSecurityMigrator.main(ZkSecurityMigrator.scala)
> Caused by: joptsimple.internal.ReflectionException: 
> java.lang.NumberFormatException: For input string: "localhost:2181"
>   at 
> joptsimple.internal.Reflection.reflectionException(Reflection.java:140)
>   at joptsimple.internal.Reflection.invoke(Reflection.java:122)
>   at 
> joptsimple.internal.MethodInvokingValueConverter.convert(MethodInvokingValueConverter.java:48)
>   at joptsimple.internal.Reflection.convertWith(Reflection.java:128)
>   at joptsimple.AbstractOptionSpec.convertWith(AbstractOptionSpec.java:90)
>   ... 6 more
> Caused by: java.lang.NumberFormatException: For input string: "localhost:2181"
>   at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>   at java.lang.Integer.parseInt(Integer.java:492)
>   at java.lang.Integer.valueOf(Integer.java:582)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at joptsimple.internal.Reflection.invoke(Reflection.java:119)
>   ... 9 more
> ===>the argument  "zookeeper.connection" has been parsed to 
> "zookeeper.connection.timeout"
> using help i found that  the argument  is :
> --zookeeper.connectSets the ZooKeeper connect string 
>  (ensemble). This parameter takes a  
>  comma-separated list of host:port   
>  pairs. (default: localhost:2181)
> --zookeeper.connection.timeout Sets the ZooKeeper connection timeout.
> the document describe wrong, and the code also has something wrong:
>  in ZkSecurityMigrator.scala,
>   val parser = new OptionParse()==>
> Any of --v, --ve, ... are accepted on the command line and treated as though 
> you had typed --verbose.
> To suppress this behavior, use the OptionParser constructor 
> OptionParser(boolean allowAbbreviations) and pass a value of false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3701) Expose KafkaStreams metrics in public API

[Eno Thereska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eno Thereska resolved KAFKA-3701.
-
Resolution: Fixed

> Expose KafkaStreams metrics in public API
> -
>
> Key: KAFKA-3701
> URL: https://issues.apache.org/jira/browse/KAFKA-3701
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Jeff Klukas
>Assignee: Eno Thereska
>Priority: Minor
>  Labels: user-experience
> Fix For: 0.10.2.0
>
>
> The Kafka clients expose their metrics registries through a `metrics` method 
> presenting an unmodifiable collection, but `KafkaStreams` does not expose its 
> registry. Currently, applications can access a StreamsMetrics instance via 
> the ProcessorContext within a Processor, but this limits flexibility.
> Having read-only access to a KafkaStreams.metrics() method would allow a 
> developer to define a health check for their application based on the metrics 
> that KafkaStreams is collecting. Or a developer might want to define a metric 
> in some other framework based on KafkaStreams' metrics.
> I am imagining that an application would build and register 
> KafkaStreams-based health checks after building a KafkaStreams instance but 
> before calling the start() method. Are metrics added to the registry at the 
> time a KafkaStreams instance is constructed, or only after calling the 
> start() method? If metrics are registered only after application startup, 
> then this approach may not be sufficient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3701) Expose KafkaStreams metrics in public API

[Eno Thereska (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820651#comment-15820651
 ] 

Eno Thereska commented on KAFKA-3701:
-

This was fixed at part of KIP-104. 

> Expose KafkaStreams metrics in public API
> -
>
> Key: KAFKA-3701
> URL: https://issues.apache.org/jira/browse/KAFKA-3701
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Jeff Klukas
>Assignee: Eno Thereska
>Priority: Minor
>  Labels: user-experience
> Fix For: 0.10.2.0
>
>
> The Kafka clients expose their metrics registries through a `metrics` method 
> presenting an unmodifiable collection, but `KafkaStreams` does not expose its 
> registry. Currently, applications can access a StreamsMetrics instance via 
> the ProcessorContext within a Processor, but this limits flexibility.
> Having read-only access to a KafkaStreams.metrics() method would allow a 
> developer to define a health check for their application based on the metrics 
> that KafkaStreams is collecting. Or a developer might want to define a metric 
> in some other framework based on KafkaStreams' metrics.
> I am imagining that an application would build and register 
> KafkaStreams-based health checks after building a KafkaStreams instance but 
> before calling the start() method. Are metrics added to the registry at the 
> time a KafkaStreams instance is constructed, or only after calling the 
> start() method? If metrics are registered only after application startup, 
> then this approach may not be sufficient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-106 - Default unclean.leader.election.enabled True => False

[Rajini Sivaram]

+1 (non-binding)

On Wed, Jan 11, 2017 at 11:15 PM, Tom Crayford  wrote:

> +1 (non-binding)
>
> On Wed, Jan 11, 2017 at 11:12 PM, Stevo Slavić  wrote:
>
> > +1 (non-binding)
> >
> > On Thu, Jan 12, 2017 at 12:11 AM, Guozhang Wang 
> > wrote:
> >
> > > +1
> > >
> > > On Wed, Jan 11, 2017 at 12:09 PM, Jeff Widman 
> wrote:
> > >
> > > > +1 nonbinding. We were bit by this in a production environment.
> > > >
> > > > On Wed, Jan 11, 2017 at 11:42 AM, Ian Wrigley 
> > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > > On Jan 11, 2017, at 11:33 AM, Jay Kreps 
> wrote:
> > > > > >
> > > > > > +1
> > > > > >
> > > > > > On Wed, Jan 11, 2017 at 10:56 AM, Ben Stopford  >
> > > > wrote:
> > > > > >
> > > > > >> Looks like there was a good consensus on the discuss thread for
> > > > KIP-106
> > > > > so
> > > > > >> lets move to a vote.
> > > > > >>
> > > > > >> Please chime in if you would like to change the default for
> > > > > >> unclean.leader.election.enabled from true to false.
> > > > > >>
> > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/%
> > > > > >> 5BWIP%5D+KIP-106+-+Change+Default+unclean.leader.
> > > > > >> election.enabled+from+True+to+False
> > > > > >>
> > > > > >> B
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: [ANNOUNCE] New committer: Grant Henke

[Rajini Sivaram]

Congratulations, Grant!

Regards,

Rajini

On Thu, Jan 12, 2017 at 8:58 AM, Damian Guy  wrote:

> Congratulations!
>
> On Thu, 12 Jan 2017 at 03:35 Jun Rao  wrote:
>
> > Grant,
> >
> > Thanks for all your contribution! Congratulations!
> >
> > Jun
> >
> > On Wed, Jan 11, 2017 at 2:51 PM, Gwen Shapira  wrote:
> >
> > > The PMC for Apache Kafka has invited Grant Henke to join as a
> > > committer and we are pleased to announce that he has accepted!
> > >
> > > Grant contributed 88 patches, 90 code reviews, countless great
> > > comments on discussions, a much-needed cleanup to our protocol and the
> > > on-going and critical work on the Admin protocol. Throughout this, he
> > > displayed great technical judgment, high-quality work and willingness
> > > to contribute where needed to make Apache Kafka awesome.
> > >
> > > Thank you for your contributions, Grant :)
> > >
> > > --
> > > Gwen Shapira
> > > Product Manager | Confluent
> > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> > > Follow us: Twitter | blog
> > >
> >
>

Re: [ANNOUNCE] New committer: Grant Henke

[Damian Guy]

Congratulations!

On Thu, 12 Jan 2017 at 03:35 Jun Rao  wrote:

> Grant,
>
> Thanks for all your contribution! Congratulations!
>
> Jun
>
> On Wed, Jan 11, 2017 at 2:51 PM, Gwen Shapira  wrote:
>
> > The PMC for Apache Kafka has invited Grant Henke to join as a
> > committer and we are pleased to announce that he has accepted!
> >
> > Grant contributed 88 patches, 90 code reviews, countless great
> > comments on discussions, a much-needed cleanup to our protocol and the
> > on-going and critical work on the Admin protocol. Throughout this, he
> > displayed great technical judgment, high-quality work and willingness
> > to contribute where needed to make Apache Kafka awesome.
> >
> > Thank you for your contributions, Grant :)
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 <(650)%20450-2760> | @gwenshap
> > Follow us: Twitter | blog
> >
>

[jira] [Resolved] (KAFKA-3714) Allow users greater access to register custom streams metrics

[Eno Thereska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eno Thereska resolved KAFKA-3714.
-
Resolution: Fixed

> Allow users greater access to register custom streams metrics
> -
>
> Key: KAFKA-3714
> URL: https://issues.apache.org/jira/browse/KAFKA-3714
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Jeff Klukas
>Assignee: Eno Thereska
>Priority: Minor
>  Labels: api
> Fix For: 0.10.3.0
>
>
> Copying in some discussion that originally appeared in 
> https://github.com/apache/kafka/pull/1362#issuecomment-219064302
> Kafka Streams is largely a higher-level abstraction on top of producers and 
> consumers, and it seems sensible to match the KafkaStreams interface to that 
> of KafkaProducer and KafkaConsumer where possible. For producers and 
> consumers, the metric registry is internal and metrics are only exposed as an 
> unmodifiable map. This allows users to access client metric values for use in 
> application health checks, etc., but doesn't allow them to register new 
> metrics.
> That approach seems reasonable if we assume that a user interested in 
> defining custom metrics is already going to be using a separate metrics 
> library. In such a case, users will likely find it easier to define metrics 
> using whatever library they're familiar with rather than learning the API for 
> Kafka's Metrics class. Is this a reasonable assumption?
> If we want to expose the Metrics instance so that users can define arbitrary 
> metrics, I'd argue that there's need for documentation updates. In 
> particular, I find the notion of metric tags confusing. Tags can be defined 
> in a MetricConfig when the Metrics instance is constructed, 
> StreamsMetricsImpl is maintaining its own set of tags, and users can set tag 
> overrides.
> If a user were to get access to the Metrics instance, they would be missing 
> the tags defined in StreamsMetricsImpl. I'm imagining that users would want 
> their custom metrics to sit alongside the predefined metrics with the same 
> tags, and users shouldn't be expected to manage those additional tags 
> themselves.
> So, why are we allowing users to define their own metrics via the 
> StreamsMetrics interface in the first place? Is it that we'd like to be able 
> to provide a built-in latency metric, but the definition depends on the 
> details of the use case so there's no generic solution? That would be 
> sufficient motivation for this special case of addLatencySensor. If we want 
> to continue down that path and give users access to define a wider range of 
> custom metrics, I'd prefer to extend the StreamsMetrics interface so that 
> users can call methods on that object, automatically getting the tags 
> appropriate for that instance rather than interacting with the raw Metrics 
> instance.
> ---
> Guozhang had the following comments:
> 1) For the producer/consumer cases, all internal metrics are provided and 
> abstracted from users, and they just need to read the documentation to poll 
> whatever provided metrics that they are interested; and if they want to 
> define more metrics, they are likely to be outside the clients themselves and 
> they can use whatever methods they like, so Metrics do not need to be exposed 
> to users.
> 2) For streams, things are a bit different: users define the computational 
> logic, which becomes part of the "Streams Client" processing and may be of 
> interests to be monitored by user themselves; think of a customized processor 
> that sends an email to some address based on a condition, and users want to 
> monitor the average rate of emails sent. Hence it is worth considering 
> whether or not they should be able to access the Metrics instance to define 
> their own along side the pre-defined metrics provided by the library.
> 3) Now, since the Metrics class was not previously designed for public usage, 
> it is not designed to be very user-friendly for defining sensors, especially 
> the semantics differences between name / scope / tags. StreamsMetrics tries 
> to hide some of these semantics confusion from users, but it still expose 
> tags and hence is not perfect in doing so. We need to think of a better 
> approach so that: 1) user defined metrics will be "aligned" (i.e. with the 
> same name prefix within a single application, with similar scope hierarchy 
> definition, etc) with library provided metrics, 2) natural APIs to do so.
> I do not have concrete ideas about 3) above on top of my head, comments are 
> more than welcomed.
> ---
> I'm not sure that I agree that 1) and 2) are truly different situations. A 
> user might choose to send email messages within a bare consumer rather than a 
> streams application, and 

[jira] [Commented] (KAFKA-3714) Allow users greater access to register custom streams metrics

[Eno Thereska (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820484#comment-15820484
 ] 

Eno Thereska commented on KAFKA-3714:
-

This was addressed in https://github.com/apache/kafka/pull/1446

> Allow users greater access to register custom streams metrics
> -
>
> Key: KAFKA-3714
> URL: https://issues.apache.org/jira/browse/KAFKA-3714
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Jeff Klukas
>Assignee: Eno Thereska
>Priority: Minor
>  Labels: api
> Fix For: 0.10.3.0
>
>
> Copying in some discussion that originally appeared in 
> https://github.com/apache/kafka/pull/1362#issuecomment-219064302
> Kafka Streams is largely a higher-level abstraction on top of producers and 
> consumers, and it seems sensible to match the KafkaStreams interface to that 
> of KafkaProducer and KafkaConsumer where possible. For producers and 
> consumers, the metric registry is internal and metrics are only exposed as an 
> unmodifiable map. This allows users to access client metric values for use in 
> application health checks, etc., but doesn't allow them to register new 
> metrics.
> That approach seems reasonable if we assume that a user interested in 
> defining custom metrics is already going to be using a separate metrics 
> library. In such a case, users will likely find it easier to define metrics 
> using whatever library they're familiar with rather than learning the API for 
> Kafka's Metrics class. Is this a reasonable assumption?
> If we want to expose the Metrics instance so that users can define arbitrary 
> metrics, I'd argue that there's need for documentation updates. In 
> particular, I find the notion of metric tags confusing. Tags can be defined 
> in a MetricConfig when the Metrics instance is constructed, 
> StreamsMetricsImpl is maintaining its own set of tags, and users can set tag 
> overrides.
> If a user were to get access to the Metrics instance, they would be missing 
> the tags defined in StreamsMetricsImpl. I'm imagining that users would want 
> their custom metrics to sit alongside the predefined metrics with the same 
> tags, and users shouldn't be expected to manage those additional tags 
> themselves.
> So, why are we allowing users to define their own metrics via the 
> StreamsMetrics interface in the first place? Is it that we'd like to be able 
> to provide a built-in latency metric, but the definition depends on the 
> details of the use case so there's no generic solution? That would be 
> sufficient motivation for this special case of addLatencySensor. If we want 
> to continue down that path and give users access to define a wider range of 
> custom metrics, I'd prefer to extend the StreamsMetrics interface so that 
> users can call methods on that object, automatically getting the tags 
> appropriate for that instance rather than interacting with the raw Metrics 
> instance.
> ---
> Guozhang had the following comments:
> 1) For the producer/consumer cases, all internal metrics are provided and 
> abstracted from users, and they just need to read the documentation to poll 
> whatever provided metrics that they are interested; and if they want to 
> define more metrics, they are likely to be outside the clients themselves and 
> they can use whatever methods they like, so Metrics do not need to be exposed 
> to users.
> 2) For streams, things are a bit different: users define the computational 
> logic, which becomes part of the "Streams Client" processing and may be of 
> interests to be monitored by user themselves; think of a customized processor 
> that sends an email to some address based on a condition, and users want to 
> monitor the average rate of emails sent. Hence it is worth considering 
> whether or not they should be able to access the Metrics instance to define 
> their own along side the pre-defined metrics provided by the library.
> 3) Now, since the Metrics class was not previously designed for public usage, 
> it is not designed to be very user-friendly for defining sensors, especially 
> the semantics differences between name / scope / tags. StreamsMetrics tries 
> to hide some of these semantics confusion from users, but it still expose 
> tags and hence is not perfect in doing so. We need to think of a better 
> approach so that: 1) user defined metrics will be "aligned" (i.e. with the 
> same name prefix within a single application, with similar scope hierarchy 
> definition, etc) with library provided metrics, 2) natural APIs to do so.
> I do not have concrete ideas about 3) above on top of my head, comments are 
> more than welcomed.
> ---
> I'm not sure that I agree that 1) and 2) are truly different situations. A 
> user might choose to send 

[jira] [Comment Edited] (KAFKA-3714) Allow users greater access to register custom streams metrics

[Eno Thereska (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820484#comment-15820484
 ] 

Eno Thereska edited comment on KAFKA-3714 at 1/12/17 8:36 AM:
--

This was addressed in https://github.com/apache/kafka/pull/1446 and KIP-104


was (Author: enothereska):
This was addressed in https://github.com/apache/kafka/pull/1446

> Allow users greater access to register custom streams metrics
> -
>
> Key: KAFKA-3714
> URL: https://issues.apache.org/jira/browse/KAFKA-3714
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Jeff Klukas
>Assignee: Eno Thereska
>Priority: Minor
>  Labels: api
> Fix For: 0.10.3.0
>
>
> Copying in some discussion that originally appeared in 
> https://github.com/apache/kafka/pull/1362#issuecomment-219064302
> Kafka Streams is largely a higher-level abstraction on top of producers and 
> consumers, and it seems sensible to match the KafkaStreams interface to that 
> of KafkaProducer and KafkaConsumer where possible. For producers and 
> consumers, the metric registry is internal and metrics are only exposed as an 
> unmodifiable map. This allows users to access client metric values for use in 
> application health checks, etc., but doesn't allow them to register new 
> metrics.
> That approach seems reasonable if we assume that a user interested in 
> defining custom metrics is already going to be using a separate metrics 
> library. In such a case, users will likely find it easier to define metrics 
> using whatever library they're familiar with rather than learning the API for 
> Kafka's Metrics class. Is this a reasonable assumption?
> If we want to expose the Metrics instance so that users can define arbitrary 
> metrics, I'd argue that there's need for documentation updates. In 
> particular, I find the notion of metric tags confusing. Tags can be defined 
> in a MetricConfig when the Metrics instance is constructed, 
> StreamsMetricsImpl is maintaining its own set of tags, and users can set tag 
> overrides.
> If a user were to get access to the Metrics instance, they would be missing 
> the tags defined in StreamsMetricsImpl. I'm imagining that users would want 
> their custom metrics to sit alongside the predefined metrics with the same 
> tags, and users shouldn't be expected to manage those additional tags 
> themselves.
> So, why are we allowing users to define their own metrics via the 
> StreamsMetrics interface in the first place? Is it that we'd like to be able 
> to provide a built-in latency metric, but the definition depends on the 
> details of the use case so there's no generic solution? That would be 
> sufficient motivation for this special case of addLatencySensor. If we want 
> to continue down that path and give users access to define a wider range of 
> custom metrics, I'd prefer to extend the StreamsMetrics interface so that 
> users can call methods on that object, automatically getting the tags 
> appropriate for that instance rather than interacting with the raw Metrics 
> instance.
> ---
> Guozhang had the following comments:
> 1) For the producer/consumer cases, all internal metrics are provided and 
> abstracted from users, and they just need to read the documentation to poll 
> whatever provided metrics that they are interested; and if they want to 
> define more metrics, they are likely to be outside the clients themselves and 
> they can use whatever methods they like, so Metrics do not need to be exposed 
> to users.
> 2) For streams, things are a bit different: users define the computational 
> logic, which becomes part of the "Streams Client" processing and may be of 
> interests to be monitored by user themselves; think of a customized processor 
> that sends an email to some address based on a condition, and users want to 
> monitor the average rate of emails sent. Hence it is worth considering 
> whether or not they should be able to access the Metrics instance to define 
> their own along side the pre-defined metrics provided by the library.
> 3) Now, since the Metrics class was not previously designed for public usage, 
> it is not designed to be very user-friendly for defining sensors, especially 
> the semantics differences between name / scope / tags. StreamsMetrics tries 
> to hide some of these semantics confusion from users, but it still expose 
> tags and hence is not perfect in doing so. We need to think of a better 
> approach so that: 1) user defined metrics will be "aligned" (i.e. with the 
> same name prefix within a single application, with similar scope hierarchy 
> definition, etc) with library provided metrics, 2) natural APIs to do so.
> I do not have concrete ideas about 3) above on top of my