Build failed in Jenkins: kafka-0.10.1-jdk7 #62

[Apache Jenkins Server]

See 

Changes:

[jason] KAFKA-4244; Fix formatting issues in documentation

[jason] MINOR: Make documentation follow latest template

[jason] MINOR: Fix typos in documentation

[jason] MINOR: Fix table of contents and section numbers in the documentation

--
[...truncated 6277 lines...]
kafka.api.AuthorizerIntegrationTest > 
testCreatePermissionNeededToReadFromNonExistentTopic STARTED

kafka.api.AuthorizerIntegrationTest > 
testCreatePermissionNeededToReadFromNonExistentTopic PASSED

kafka.api.AuthorizerIntegrationTest > 
testPatternSubscriptionWithTopicDescribeOnlyAndGroupRead STARTED

kafka.api.AuthorizerIntegrationTest > 
testPatternSubscriptionWithTopicDescribeOnlyAndGroupRead PASSED

kafka.api.AuthorizerIntegrationTest > testDeleteWithWildCardAuth STARTED

kafka.api.AuthorizerIntegrationTest > testDeleteWithWildCardAuth PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoAccess STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoAccess PASSED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicRead STARTED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicRead PASSED

kafka.api.AuthorizerIntegrationTest > testListOffsetsWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testListOffsetsWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > 
testCreatePermissionNeededForWritingToNonExistentTopic STARTED

kafka.api.AuthorizerIntegrationTest > 
testCreatePermissionNeededForWritingToNonExistentTopic PASSED

kafka.api.AuthorizerIntegrationTest > 
testPatternSubscriptionMatchingInternalTopicWithDescribeOnlyPermission STARTED

kafka.api.AuthorizerIntegrationTest > 
testPatternSubscriptionMatchingInternalTopicWithDescribeOnlyPermission PASSED

kafka.api.AuthorizerIntegrationTest > 
testSimpleConsumeWithOffsetLookupAndNoGroupAccess STARTED

kafka.api.AuthorizerIntegrationTest > 
testSimpleConsumeWithOffsetLookupAndNoGroupAccess PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > testProduceWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testProduceWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicWrite STARTED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicWrite PASSED

kafka.api.AuthorizerIntegrationTest > testAuthorizationWithTopicNotExisting 
STARTED

kafka.api.AuthorizerIntegrationTest > testAuthorizationWithTopicNotExisting 
PASSED

kafka.api.AuthorizerIntegrationTest > testListOffsetsWithTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testListOffsetsWithTopicDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicAndGroupRead STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicAndGroupRead PASSED

kafka.api.AuthorizerIntegrationTest > 
testPatternSubscriptionNotMatchingInternalTopic STARTED

kafka.api.AuthorizerIntegrationTest > 
testPatternSubscriptionNotMatchingInternalTopic PASSED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithoutDescribe 
STARTED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithoutDescribe 
PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicWrite STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicWrite PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoGroupAccess STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoGroupAccess PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoAccess STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoAccess PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithoutTopicDescribeAccess 
STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithoutTopicDescribeAccess 
PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoGroupAccess STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoGroupAccess PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithNoAccess STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithNoAccess PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithTopicAndGroupRead 
STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithTopicAndGroupRead 
PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > 

[GitHub] kafka pull request #2010: MINOR: Fixed broken links in the documentation

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/2010

MINOR: Fixed broken links in the documentation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka doc/fix_hyperlinks

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2010.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2010


commit bf6af8f36fbce16704ab0b9cabea53f4d7f0143e
Author: Vahid Hashemian 
Date:   2016-10-11T04:52:16Z

MINOR: Fixed broken links in the documentation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #1618

[Apache Jenkins Server]

See 

Changes:

[jason] KAFKA-4244; Fix formatting issues in documentation

[jjkoshy] KAFKA-3175; Topic not accessible after deletion even when

[jason] MINOR: Make documentation follow latest template

[jason] MINOR: Fix typos in documentation

[jason] MINOR: Fix table of contents and section numbers in the documentation

--
[...truncated 14071 lines...]
org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutIfAbsent PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testRestoreWithDefaultSerdes STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testRestoreWithDefaultSerdes PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testRestore STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testRestore PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutGetRange STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutGetRange PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutGetRangeWithDefaultSerdes STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutGetRangeWithDefaultSerdes PASSED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldPeekNext STARTED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldPeekNext PASSED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldPeekAndIterate STARTED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldPeekAndIterate PASSED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldThrowNoSuchElementWhenNoMoreItemsLeftAndPeekNextCalled STARTED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldThrowNoSuchElementWhenNoMoreItemsLeftAndPeekNextCalled PASSED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldThrowNoSuchElementWhenNoMoreItemsLeftAndNextCalled STARTED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldThrowNoSuchElementWhenNoMoreItemsLeftAndNextCalled PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheWindowStoreIteratorTest
 > shouldIterateOverValueFromBothIterators STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheWindowStoreIteratorTest
 > shouldIterateOverValueFromBothIterators PASSED

org.apache.kafka.streams.state.internals.StreamThreadStateStoreProviderTest > 
shouldReturnEmptyListIfNoStoresFoundWithName STARTED

org.apache.kafka.streams.state.internals.StreamThreadStateStoreProviderTest > 
shouldReturnEmptyListIfNoStoresFoundWithName PASSED

org.apache.kafka.streams.state.internals.StreamThreadStateStoreProviderTest > 
shouldThrowInvalidStoreExceptionIfKVStoreClosed STARTED

org.apache.kafka.streams.state.internals.StreamThreadStateStoreProviderTest > 
shouldThrowInvalidStoreExceptionIfKVStoreClosed PASSED

org.apache.kafka.streams.state.internals.StreamThreadStateStoreProviderTest > 
shouldThrowInvalidStoreExceptionIfNotAllStoresAvailable STARTED

org.apache.kafka.streams.state.internals.StreamThreadStateStoreProviderTest > 
shouldThrowInvalidStoreExceptionIfNotAllStoresAvailable PASSED

org.apache.kafka.streams.state.internals.StreamThreadStateStoreProviderTest > 
shouldFindWindowStores STARTED

org.apache.kafka.streams.state.internals.StreamThreadStateStoreProviderTest > 
shouldFindWindowStores PASSED

org.apache.kafka.streams.state.internals.StreamThreadStateStoreProviderTest > 
shouldReturnEmptyListIfStoreExistsButIsNotOfTypeValueStore STARTED

org.apache.kafka.streams.state.internals.StreamThreadStateStoreProviderTest > 
shouldReturnEmptyListIfStoreExistsButIsNotOfTypeValueStore PASSED

org.apache.kafka.streams.state.internals.StreamThreadStateStoreProviderTest > 
shouldFindKeyValueStores STARTED

org.apache.kafka.streams.state.internals.StreamThreadStateStoreProviderTest > 
shouldFindKeyValueStores PASSED

org.apache.kafka.streams.state.internals.StreamThreadStateStoreProviderTest > 
shouldThrowInvalidStoreExceptionIfWindowStoreClosed STARTED

org.apache.kafka.streams.state.internals.StreamThreadStateStoreProviderTest > 
shouldThrowInvalidStoreExceptionIfWindowStoreClosed PASSED

org.apache.kafka.streams.state.StoresTest > 
shouldCreateInMemoryStoreSupplierWithLoggedConfig STARTED

org.apache.kafka.streams.state.StoresTest > 
shouldCreateInMemoryStoreSupplierWithLoggedConfig PASSED

org.apache.kafka.streams.state.StoresTest > 
shouldCreatePersistenStoreSupplierNotLogged STARTED

org.apache.kafka.streams.state.StoresTest > 
shouldCreatePersistenStoreSupplierNotLogged PASSED

org.apache.kafka.streams.state.StoresTest > 
shouldCreatePersistenStoreSupplierWithLoggedConfig 

[GitHub] kafka pull request #2009: KAFKA-4290: Fix timeout overflow in WorkerCoordina...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/2009

KAFKA-4290: Fix timeout overflow in WorkerCoordinator.poll



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-4290

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2009.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2009


commit 50c65a82923fc62b3f4b73812097bcc01aa74fef
Author: Jason Gustafson 
Date:   2016-10-11T04:27:45Z

KAFKA-4290: Fix timeout overflow in WorkerCoordinator.poll




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4290) High CPU caused by timeout overflow in WorkerCoordinator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564425#comment-15564425
 ] 

ASF GitHub Bot commented on KAFKA-4290:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/2009

KAFKA-4290: Fix timeout overflow in WorkerCoordinator.poll



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-4290

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2009.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2009


commit 50c65a82923fc62b3f4b73812097bcc01aa74fef
Author: Jason Gustafson 
Date:   2016-10-11T04:27:45Z

KAFKA-4290: Fix timeout overflow in WorkerCoordinator.poll




> High CPU caused by timeout overflow in WorkerCoordinator
> 
>
> Key: KAFKA-4290
> URL: https://issues.apache.org/jira/browse/KAFKA-4290
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.10.1.0
>
>
> The timeout passed to {{WorkerCoordinator.poll()}} can overflow if large 
> enough because we add it to the current time in order to calculate the call's 
> deadline. This shortcuts the poll loop and results in a very tight event loop 
> which can saturate a CPU. We hit this case out of the box because Connect 
> uses a default timeout of {{Long.MAX_VALUE}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #965

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-eu2 (Ubuntu ubuntu ubuntu-eu docker) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
ERROR: Workspace has a .git repository, but it appears to be corrupt.
hudson.plugins.git.GitException: Command "git rev-parse --is-inside-work-tree" 
returned status code 128:
stdout: 
stderr: fatal: Not a git repository (or any of the parent directories): .git

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1723)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1699)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1695)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1317)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1329)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.hasGitRepo(CliGitAPIImpl.java:195)
at hudson.plugins.git.GitAPI.hasGitRepo(GitAPI.java:232)
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:884)
at 
hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:859)
at 
hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:818)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Cloning the remote Git repository
Cloning repository https://git-wip-us.apache.org/repos/asf/kafka.git
 > git init  # timeout=10
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Could not init 

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:652)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:463)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ..remote call to ubuntu-eu2(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:781)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
at sun.reflect.GeneratedMethodAccessor526.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
at com.sun.proxy.$Proxy177.execute(Unknown Source)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1046)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1086)
at hudson.scm.SCM.checkout(SCM.java:485)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 

[jira] [Created] (KAFKA-4290) High CPU caused by timeout overflow in WorkerCoordinator

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-4290:
--

 Summary: High CPU caused by timeout overflow in WorkerCoordinator
 Key: KAFKA-4290
 URL: https://issues.apache.org/jira/browse/KAFKA-4290
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Jason Gustafson
Assignee: Jason Gustafson
Priority: Blocker
 Fix For: 0.10.1.0


The timeout passed to {{WorkerCoordinator.poll()}} can overflow if large enough 
because we add it to the current time in order to calculate the call's 
deadline. This shortcuts the poll loop and results in a very tight event loop 
which can saturate a CPU. We hit this case out of the box because Connect uses 
a default timeout of {{Long.MAX_VALUE}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] 0.10.1.0 RC1

[Jason Gustafson]

The documentation is mostly fixed now: http://kafka.apache.org/
0101/documentation.html. Thanks to Derrick Or for all the help. Let me know
if anyone notices any additional problems.

-Jason

On Mon, Oct 10, 2016 at 1:10 PM, Jason Gustafson  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 0.10.1.0. This is
> a minor release that includes great new features including throttled
> replication, secure quotas, time-based log searching, and queryable state
> for Kafka Streams. A full list of the content can be found here:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.1.
>
> One quick note on the docs. Because of all the recent improvements, the
> documentation is still a bit out of sync with what's visible on the Kafka
> homepage. This should be fixed soon (definitely before the release is
> finalized).
>
> Release notes for the 0.10.1.0 release:
> http://home.apache.org/~jgus/kafka-0.10.1.0-rc1/RELEASE_NOTES.html
> 
>
> *** Please download, test and vote by Thursday, Oct 13, 1pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~jgus/kafka-0.10.1.0-rc1/
> 
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~jgus/kafka-0.10.1.0-rc1/javadoc/
> 
>
> * Tag to be voted upon (off 0.10.1 branch) is the 0.10.1.0-rc1 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> 6eda15a97ffe17d636c390c0e0b28c8349993941
>
> * Documentation:
> http://kafka.apache.org/0101/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0101/protocol.html
>
> * Tests:
> Unit tests: https://builds.apache.org/job/kafka-0.10.1-jdk7/59/
> System tests: http://testing.confluent.io/confluent-kafka-0-10-1-system-
> test-results/?prefix=2016-10-10--001.1476110532--apache--0.10.1--e696f17/
>
> Thanks,
>
> Jason
>

Build failed in Jenkins: kafka-trunk-jdk8 #964

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-eu2 (Ubuntu ubuntu ubuntu-eu docker) in workspace 

Cloning the remote Git repository
Cloning repository https://git-wip-us.apache.org/repos/asf/kafka.git
 > git init  # timeout=10
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Could not init 

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:652)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:463)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ..remote call to ubuntu-eu2(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:781)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
at sun.reflect.GeneratedMethodAccessor526.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
at com.sun.proxy.$Proxy177.execute(Unknown Source)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1046)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1086)
at hudson.scm.SCM.checkout(SCM.java:485)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1741)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Caused by: hudson.plugins.git.GitException: Command "git init 
 returned status code 128:
stdout: 
stderr: error: copy-fd: write returned No space left on device
fatal: cannot copy '/usr/share/git-core/templates/hooks/update.sample' to 
': 
No space left on device

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1723)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1699)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1695)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1317)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:650)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:463)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at 

[jira] [Commented] (KAFKA-4010) ConfigDef.toRst() should create sections for each group

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564271#comment-15564271
 ] 

ASF GitHub Bot commented on KAFKA-4010:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1964


> ConfigDef.toRst() should create sections for each group
> ---
>
> Key: KAFKA-4010
> URL: https://issues.apache.org/jira/browse/KAFKA-4010
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Shikhar Bhushan
>Assignee: Rekha Joshi
>Priority: Minor
> Fix For: 0.10.2.0
>
>
> Currently the ordering seems a bit arbitrary. There is a logical grouping 
> that connectors are now able to specify with the 'group' field, which we 
> should use as section headers. Also it would be good to generate {{:ref:}} 
> for each section.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4010) ConfigDef.toRst() should create sections for each group

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-4010.
--
   Resolution: Fixed
Fix Version/s: 0.10.2.0

Issue resolved by pull request 1964
[https://github.com/apache/kafka/pull/1964]

> ConfigDef.toRst() should create sections for each group
> ---
>
> Key: KAFKA-4010
> URL: https://issues.apache.org/jira/browse/KAFKA-4010
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Shikhar Bhushan
>Assignee: Rekha Joshi
>Priority: Minor
> Fix For: 0.10.2.0
>
>
> Currently the ordering seems a bit arbitrary. There is a logical grouping 
> that connectors are now able to specify with the 'group' field, which we 
> should use as section headers. Also it would be good to generate {{:ref:}} 
> for each section.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1964: KAFKA-4010: add ConfigDef toEnrichedRst() for addi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1964


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2008: MINOR: Add images missing from documentation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2008


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2008: MINOR: Add images missing from documentation

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/2008

MINOR: Add images missing from documentation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka add-missing-images

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2008.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2008


commit fbd4f14bcdb1fe0809feba9844f84ef0dada9af6
Author: Jason Gustafson 
Date:   2016-10-11T02:52:58Z

MINOR: Add images missing from documentation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka-site issue #24: update htaccess to load images nested inside of doc fo...

[hachikuji]

Github user hachikuji commented on the issue:

https://github.com/apache/kafka-site/pull/24
  
LGTM. Thanks for the fixes!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk8 #963

[Apache Jenkins Server]

See 

Changes:

[jason] KAFKA-4244; Fix formatting issues in documentation

[jjkoshy] KAFKA-3175; Topic not accessible after deletion even when

[jason] MINOR: Make documentation follow latest template

[jason] MINOR: Fix typos in documentation

[jason] MINOR: Fix table of contents and section numbers in the documentation

--
[...truncated 6257 lines...]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

kafka.api.SslProducerSendTest > testSendOffset PASSED

kafka.api.SslProducerSendTest > testSendCompressedMessageWithCreateTime STARTED

kafka.api.SslProducerSendTest > testSendCompressedMessageWithCreateTime PASSED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromCallerThread STARTED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromCallerThread PASSED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromSenderThread STARTED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromSenderThread PASSED

kafka.api.SaslPlainPlaintextConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslPlainPlaintextConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslPlainPlaintextConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslPlainPlaintextConsumerTest > testSimpleConsumption PASSED

kafka.api.SaslPlaintextConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslPlaintextConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslPlaintextConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslPlaintextConsumerTest > testSimpleConsumption PASSED

kafka.api.EndToEndClusterIdTest > testEndToEnd STARTED

kafka.api.EndToEndClusterIdTest > testEndToEnd PASSED

kafka.api.SaslSslConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslSslConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslSslConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslSslConsumerTest > testSimpleConsumption PASSED

kafka.api.RequestResponseSerializationTest > 
testSerializationAndDeserialization STARTED

kafka.api.RequestResponseSerializationTest > 
testSerializationAndDeserialization PASSED

kafka.api.RequestResponseSerializationTest > testFetchResponseVersion STARTED

kafka.api.RequestResponseSerializationTest > testFetchResponseVersion PASSED

kafka.api.RequestResponseSerializationTest > testProduceResponseVersion STARTED

kafka.api.RequestResponseSerializationTest > testProduceResponseVersion PASSED

kafka.api.AdminClientTest > testDescribeGroup STARTED

kafka.api.AdminClientTest > testDescribeGroup PASSED

kafka.api.AdminClientTest > testDescribeConsumerGroup STARTED

kafka.api.AdminClientTest > testDescribeConsumerGroup PASSED

kafka.api.AdminClientTest > testListGroups STARTED

kafka.api.AdminClientTest > testListGroups PASSED

kafka.api.AdminClientTest > testDescribeConsumerGroupForNonExistentGroup STARTED

kafka.api.AdminClientTest > testDescribeConsumerGroupForNonExistentGroup PASSED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SslEndToEndAuthorizationTest > testProduceConsumeViaAssign STARTED

kafka.api.SslEndToEndAuthorizationTest > testProduceConsumeViaAssign PASSED

kafka.api.SslEndToEndAuthorizationTest > testNoConsumeWithDescribeAclViaAssign 
STARTED

kafka.api.SslEndToEndAuthorizationTest > testNoConsumeWithDescribeAclViaAssign 
PASSED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.SslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.SslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl STARTED

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl PASSED

kafka.api.SslEndToEndAuthorizationTest > testProduceConsumeViaSubscribe STARTED

kafka.api.SslEndToEndAuthorizationTest > testProduceConsumeViaSubscribe PASSED

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithoutDescribeAcl STARTED

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithoutDescribeAcl PASSED

kafka.api.RackAwareAutoTopicCreationTest > testAutoCreateTopic STARTED

kafka.api.RackAwareAutoTopicCreationTest > testAutoCreateTopic PASSED

kafka.api.PlaintextConsumerTest > testEarliestOrLatestOffsets STARTED

kafka.api.PlaintextConsumerTest > testEarliestOrLatestOffsets PASSED

kafka.api.PlaintextConsumerTest > testPartitionsForAutoCreate STARTED

kafka.api.PlaintextConsumerTest > testPartitionsForAutoCreate 

Build failed in Jenkins: kafka-0.10.1-jdk7 #61

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-eu2 (Ubuntu ubuntu ubuntu-eu docker) in workspace 

java.io.IOException: Failed to mkdirs: 

at hudson.FilePath.mkdirs(FilePath.java:1191)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1267)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1741)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Retrying after 10 seconds
java.io.IOException: Failed to mkdirs: 

at hudson.FilePath.mkdirs(FilePath.java:1191)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1267)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1741)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Retrying after 10 seconds
java.io.IOException: Failed to mkdirs: 

at hudson.FilePath.mkdirs(FilePath.java:1191)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1267)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1741)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Recording test results
ERROR: Build step failed with exception
 does not exist.
at 
org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:483)
at 
org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:460)
at 
hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:127)
at 
hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:107)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ..remote call to ubuntu-eu2(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:781)
at hudson.FilePath.act(FilePath.java:1007)
at hudson.FilePath.act(FilePath.java:996)
at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:103)
at 
hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:128)
at 
hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:149)
at 
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:720)
at hudson.model.Build$BuildExecution.post2(Build.java:185)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:665)
at 

[jira] [Updated] (KAFKA-4114) Allow for different "auto.offset.reset" strategies for different input streams

[Bill Bejeck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Bejeck updated KAFKA-4114:
---
Status: Patch Available  (was: In Progress)

> Allow for different "auto.offset.reset" strategies for different input streams
> --
>
> Key: KAFKA-4114
> URL: https://issues.apache.org/jira/browse/KAFKA-4114
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Bill Bejeck
>
> Today we only have one consumer config "offset.auto.reset" to control that 
> behavior, which means all streams are read either from "earliest" or "latest".
> However, it would be useful to improve this settings to allow users have 
> finer control over different input stream. For example, with two input 
> streams, one of them always reading from offset 0 upon (re)-starting, and the 
> other reading for log end offset.
> This JIRA requires to extend {{KStreamBuilder}} API for methods 
> {{.stream(...)}} and {{.table(...)}} to add a new parameter that indicate the 
> initial offset to be used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4114) Allow for different "auto.offset.reset" strategies for different input streams

[Bill Bejeck (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564050#comment-15564050
 ] 

Bill Bejeck commented on KAFKA-4114:


Fair enough doing so now.

> Allow for different "auto.offset.reset" strategies for different input streams
> --
>
> Key: KAFKA-4114
> URL: https://issues.apache.org/jira/browse/KAFKA-4114
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Bill Bejeck
>
> Today we only have one consumer config "offset.auto.reset" to control that 
> behavior, which means all streams are read either from "earliest" or "latest".
> However, it would be useful to improve this settings to allow users have 
> finer control over different input stream. For example, with two input 
> streams, one of them always reading from offset 0 upon (re)-starting, and the 
> other reading for log end offset.
> This JIRA requires to extend {{KStreamBuilder}} API for methods 
> {{.stream(...)}} and {{.table(...)}} to add a new parameter that indicate the 
> initial offset to be used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2007: Kafka 4114 allow different offset reset strategies

[bbejeck]

GitHub user bbejeck opened a pull request:

https://github.com/apache/kafka/pull/2007

Kafka 4114 allow different offset reset strategies

@mjsax 

Here's my first pass at finer grained auto offset reset strategies.

I've left TODO comments about whether we want to consider adding this to 
`KGroupedTable.aggregate` and `KStreamImpl` when re-partitioning a source.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bbejeck/kafka 
KAFKA-4114_allow_different_offset_reset_strategies

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2007.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2007


commit 7f96f19894c617c6f4568202511431ad2a1551f3
Author: bbejeck 
Date:   2016-10-06T01:40:13Z

KAFKA-4114 : initial work on different auto offset reset strategies for 
different input streams

commit eec398a960b89840dca7e3d24bc563de5a96faa1
Author: bbejeck 
Date:   2016-10-08T01:45:09Z

KAFKA-4114 : added additional overloaded methods to TopologyBuilder, 
cleaned up test, reverted KGroupedTableImpl, KStreamImpl and 
ProcessorTopologyTest to orignal state with new overloaded methods




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka-site issue #24: update htaccess to load images nested inside of doc fo...

[derrickdoo]

Github user derrickdoo commented on the issue:

https://github.com/apache/kafka-site/pull/24
  
- stop displaying code elements as display:block
- don't rewrite requests for images nested in the specific doc version 
directories


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka-site pull request #24: update htaccess to load images nested inside of...

[derrickdoo]

GitHub user derrickdoo opened a pull request:

https://github.com/apache/kafka-site/pull/24

update htaccess to load images nested inside of doc folders



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/derrickdoo/kafka-site imagePaths

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka-site/pull/24.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #24


commit 6f8bec72e55b57049ce643403d650d084de26d08
Author: Derrick Or 
Date:   2016-10-11T00:24:41Z

update htaccess to load images nested inside of doc folders




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4289) CPU wasted on reflection calls initializing short-lived loggers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563996#comment-15563996
 ] 

ASF GitHub Bot commented on KAFKA-4289:
---

GitHub user radai-rosenblatt opened a pull request:

https://github.com/apache/kafka/pull/2006

KAFKA-4289 - moved short-lived loggers to companion objects

Signed-off-by: radai-rosenblatt 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/radai-rosenblatt/kafka omgsrsly

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2006.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2006


commit 7b475e6eaffab891b8737c00df71d951cc52c81b
Author: radai-rosenblatt 
Date:   2016-10-11T00:21:07Z

KAFKA-4289 - moved short-lived loggers to companion objects

Signed-off-by: radai-rosenblatt 




> CPU wasted on reflection calls initializing short-lived loggers
> ---
>
> Key: KAFKA-4289
> URL: https://issues.apache.org/jira/browse/KAFKA-4289
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: radai rosenblatt
>
> an internal profiling run at linkedin found ~5% of the CPU time consumed by 
> `sun.reflect.Reflection.getCallerClass()`.
> digging into the stack trace shows its from initializing short lived logger 
> objects in `FileMessageSet` and `RequestChannel.Request`



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4289) CPU wasted on reflection calls initializing short-lived loggers

[radai rosenblatt (JIRA)]

radai rosenblatt created KAFKA-4289:
---

 Summary: CPU wasted on reflection calls initializing short-lived 
loggers
 Key: KAFKA-4289
 URL: https://issues.apache.org/jira/browse/KAFKA-4289
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.0.1
Reporter: radai rosenblatt


an internal profiling run at linkedin found ~5% of the CPU time consumed by 
`sun.reflect.Reflection.getCallerClass()`.

digging into the stack trace shows its from initializing short lived logger 
objects in `FileMessageSet` and `RequestChannel.Request`



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2006: KAFKA-4289 - moved short-lived loggers to companio...

[radai-rosenblatt]

GitHub user radai-rosenblatt opened a pull request:

https://github.com/apache/kafka/pull/2006

KAFKA-4289 - moved short-lived loggers to companion objects

Signed-off-by: radai-rosenblatt 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/radai-rosenblatt/kafka omgsrsly

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2006.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2006


commit 7b475e6eaffab891b8737c00df71d951cc52c81b
Author: radai-rosenblatt 
Date:   2016-10-11T00:21:07Z

KAFKA-4289 - moved short-lived loggers to companion objects

Signed-off-by: radai-rosenblatt 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-4288) Add an "Important Server Configs" section to documentation

[Vahid Hashemian (JIRA)]

Vahid Hashemian created KAFKA-4288:
--

 Summary: Add an "Important Server Configs" section to documentation
 Key: KAFKA-4288
 URL: https://issues.apache.org/jira/browse/KAFKA-4288
 Project: Kafka
  Issue Type: Improvement
  Components: documentation
Reporter: Vahid Hashemian


There is an {{Important Client Configs}} section under {{6. Operations / 6.3 
Important Configs}} in the documentation. It would be useful and educational to 
have an {{Important Server Configs}} section there too.

Note, a link to {{Important Server Configs}} used to exist in the documentation 
that did not point to any existing content, and [was 
removed|https://github.com/apache/kafka/pull/2004] in 0.10.1.0. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2004: MINOR: Fix table of content and section numbers in...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2004


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4114) Allow for different "auto.offset.reset" strategies for different input streams

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563941#comment-15563941
 ] 

Matthias J. Sax commented on KAFKA-4114:


Just open an PR whenever you are ready. :)

> Allow for different "auto.offset.reset" strategies for different input streams
> --
>
> Key: KAFKA-4114
> URL: https://issues.apache.org/jira/browse/KAFKA-4114
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Bill Bejeck
>
> Today we only have one consumer config "offset.auto.reset" to control that 
> behavior, which means all streams are read either from "earliest" or "latest".
> However, it would be useful to improve this settings to allow users have 
> finer control over different input stream. For example, with two input 
> streams, one of them always reading from offset 0 upon (re)-starting, and the 
> other reading for log end offset.
> This JIRA requires to extend {{KStreamBuilder}} API for methods 
> {{.stream(...)}} and {{.table(...)}} to add a new parameter that indicate the 
> initial offset to be used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4114) Allow for different "auto.offset.reset" strategies for different input streams

[Bill Bejeck (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563934#comment-15563934
 ] 

Bill Bejeck commented on KAFKA-4114:


I have a working solution, just waiting for resolution to KAFKA-4269 so I can 
merge those changes into the PR for this Jira ticket.

> Allow for different "auto.offset.reset" strategies for different input streams
> --
>
> Key: KAFKA-4114
> URL: https://issues.apache.org/jira/browse/KAFKA-4114
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Bill Bejeck
>
> Today we only have one consumer config "offset.auto.reset" to control that 
> behavior, which means all streams are read either from "earliest" or "latest".
> However, it would be useful to improve this settings to allow users have 
> finer control over different input stream. For example, with two input 
> streams, one of them always reading from offset 0 upon (re)-starting, and the 
> other reading for log end offset.
> This JIRA requires to extend {{KStreamBuilder}} API for methods 
> {{.stream(...)}} and {{.table(...)}} to add a new parameter that indicate the 
> initial offset to be used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4269) Multiple KStream instances with at least one Regex source causes NPE when using multiple consumers

[Bill Bejeck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Bejeck updated KAFKA-4269:
---
Fix Version/s: 0.10.1.0
   Status: Patch Available  (was: In Progress)

> Multiple KStream instances with at least one Regex source causes NPE when 
> using multiple consumers
> --
>
> Key: KAFKA-4269
> URL: https://issues.apache.org/jira/browse/KAFKA-4269
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
> Fix For: 0.10.1.0
>
>
> I discovered this issue while doing testing for for KAFKA-4114. 
> KAFKA-4131 fixed the issue of a _single_ KStream with a regex source on 
> partitioned topics across multiple consumers.
> //KAFKA-4131 fixed this case assuming an "foo*" topics are partitioned
> KStream kstream = builder.source(Pattern.compile("foo.*"));
> KafkaStream stream = new KafkaStreams(builder, props);
> stream.start();  
> This is a new issue where there are _multiple_
> KStream instances (and one has a regex source) within a single KafkaStreams 
> object. When running the second or "following"
> consumer there are NPE errors generated in the RecordQueue.addRawRecords 
> method when attempting to consume records. 
> For example:
> KStream kstream = builder.source(Pattern.compile("foo.*"));
> KStream kstream2 = builder.source(.): //can be regex or named topic 
> sources
> KafkaStream stream = new KafkaStreams(builder, props);
> stream.start();
> By adding an additional KStream instance like above (whether Regex or Named 
> topic) causes a NPE when run as "follower"
> From my initial debugging I can see the TopicPartition assignments being set 
> on the "follower" KafkaStreams instance, but need to track down why and where 
> all assignments aren't being set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4269) Multiple KStream instances with at least one Regex source causes NPE when using multiple consumers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563913#comment-15563913
 ] 

ASF GitHub Bot commented on KAFKA-4269:
---

GitHub user bbejeck opened a pull request:

https://github.com/apache/kafka/pull/2005

KAFKA-4269 extracted code updating topics when regex pattern specifie…

…d out of topicGroups method. The topicGroups method only called from 
StreamPartitionAssignor when KafkaStreams object  is the leader, needs to be 
executed for clients.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bbejeck/kafka 
KAFKA-4269_multiple_kstream_instances_mult_consumers_npe

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2005.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2005


commit 58c694e14fd4ba5ee134be54e3f75f4aafa8a1d6
Author: bbejeck 
Date:   2016-10-10T23:26:24Z

KAFKA-4269 extracted code updating topics when regex pattern specified out 
of topicGroups method. The topicGroups method only called from 
StreamPartitionAssignor when KafkaStreams object  is the leader, needs to be 
executed for clients.




> Multiple KStream instances with at least one Regex source causes NPE when 
> using multiple consumers
> --
>
> Key: KAFKA-4269
> URL: https://issues.apache.org/jira/browse/KAFKA-4269
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
>
> I discovered this issue while doing testing for for KAFKA-4114. 
> KAFKA-4131 fixed the issue of a _single_ KStream with a regex source on 
> partitioned topics across multiple consumers.
> //KAFKA-4131 fixed this case assuming an "foo*" topics are partitioned
> KStream kstream = builder.source(Pattern.compile("foo.*"));
> KafkaStream stream = new KafkaStreams(builder, props);
> stream.start();  
> This is a new issue where there are _multiple_
> KStream instances (and one has a regex source) within a single KafkaStreams 
> object. When running the second or "following"
> consumer there are NPE errors generated in the RecordQueue.addRawRecords 
> method when attempting to consume records. 
> For example:
> KStream kstream = builder.source(Pattern.compile("foo.*"));
> KStream kstream2 = builder.source(.): //can be regex or named topic 
> sources
> KafkaStream stream = new KafkaStreams(builder, props);
> stream.start();
> By adding an additional KStream instance like above (whether Regex or Named 
> topic) causes a NPE when run as "follower"
> From my initial debugging I can see the TopicPartition assignments being set 
> on the "follower" KafkaStreams instance, but need to track down why and where 
> all assignments aren't being set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2005: KAFKA-4269 extracted code updating topics when reg...

[bbejeck]

GitHub user bbejeck opened a pull request:

https://github.com/apache/kafka/pull/2005

KAFKA-4269 extracted code updating topics when regex pattern specifie…

…d out of topicGroups method. The topicGroups method only called from 
StreamPartitionAssignor when KafkaStreams object  is the leader, needs to be 
executed for clients.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bbejeck/kafka 
KAFKA-4269_multiple_kstream_instances_mult_consumers_npe

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2005.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2005


commit 58c694e14fd4ba5ee134be54e3f75f4aafa8a1d6
Author: bbejeck 
Date:   2016-10-10T23:26:24Z

KAFKA-4269 extracted code updating topics when regex pattern specified out 
of topicGroups method. The topicGroups method only called from 
StreamPartitionAssignor when KafkaStreams object  is the leader, needs to be 
executed for clients.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk8 #961

[Apache Jenkins Server]

See 

Changes:

[jason] MINOR: Fixed introduction doc - wrong streams api link

--
[...truncated 6680 lines...]

kafka.api.PlaintextConsumerTest > testMultiConsumerDefaultAssignment PASSED

kafka.api.PlaintextConsumerTest > testAutoCommitOnClose STARTED

kafka.api.PlaintextConsumerTest > testAutoCommitOnClose PASSED

kafka.api.PlaintextConsumerTest > testListTopics STARTED

kafka.api.PlaintextConsumerTest > testListTopics PASSED

kafka.api.PlaintextConsumerTest > testExpandingTopicSubscriptions STARTED

kafka.api.PlaintextConsumerTest > testExpandingTopicSubscriptions PASSED

kafka.api.PlaintextConsumerTest > testInterceptors STARTED

kafka.api.PlaintextConsumerTest > testInterceptors PASSED

kafka.api.PlaintextConsumerTest > testPatternUnsubscription STARTED

kafka.api.PlaintextConsumerTest > testPatternUnsubscription PASSED

kafka.api.PlaintextConsumerTest > testGroupConsumption STARTED

kafka.api.PlaintextConsumerTest > testGroupConsumption PASSED

kafka.api.PlaintextConsumerTest > testPartitionsFor STARTED

kafka.api.PlaintextConsumerTest > testPartitionsFor PASSED

kafka.api.PlaintextConsumerTest > testAutoCommitOnRebalance STARTED

kafka.api.PlaintextConsumerTest > testAutoCommitOnRebalance PASSED

kafka.api.PlaintextConsumerTest > testInterceptorsWithWrongKeyValue STARTED

kafka.api.PlaintextConsumerTest > testInterceptorsWithWrongKeyValue PASSED

kafka.api.PlaintextConsumerTest > testMaxPollIntervalMsDelayInAssignment STARTED

kafka.api.PlaintextConsumerTest > testMaxPollIntervalMsDelayInAssignment PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerRoundRobinAssignment STARTED

kafka.api.PlaintextConsumerTest > testMultiConsumerRoundRobinAssignment PASSED

kafka.api.PlaintextConsumerTest > testPartitionPauseAndResume STARTED

kafka.api.PlaintextConsumerTest > testPartitionPauseAndResume PASSED

kafka.api.PlaintextConsumerTest > testConsumeMessagesWithLogAppendTime STARTED

kafka.api.PlaintextConsumerTest > testConsumeMessagesWithLogAppendTime PASSED

kafka.api.PlaintextConsumerTest > testAutoCommitOnCloseAfterWakeup STARTED

kafka.api.PlaintextConsumerTest > testAutoCommitOnCloseAfterWakeup PASSED

kafka.api.PlaintextConsumerTest > testMaxPollRecords STARTED

kafka.api.PlaintextConsumerTest > testMaxPollRecords PASSED

kafka.api.PlaintextConsumerTest > testAutoOffsetReset STARTED

kafka.api.PlaintextConsumerTest > testAutoOffsetReset PASSED

kafka.api.PlaintextConsumerTest > testFetchInvalidOffset STARTED

kafka.api.PlaintextConsumerTest > testFetchInvalidOffset PASSED

kafka.api.PlaintextConsumerTest > testAutoCommitIntercept STARTED

kafka.api.PlaintextConsumerTest > testAutoCommitIntercept PASSED

kafka.api.PlaintextConsumerTest > 
testFetchHonoursMaxPartitionFetchBytesIfLargeRecordNotFirst STARTED

kafka.api.PlaintextConsumerTest > 
testFetchHonoursMaxPartitionFetchBytesIfLargeRecordNotFirst PASSED

kafka.api.PlaintextConsumerTest > testCommitSpecifiedOffsets STARTED

kafka.api.PlaintextConsumerTest > testCommitSpecifiedOffsets PASSED

kafka.api.PlaintextConsumerTest > testCommitMetadata STARTED

kafka.api.PlaintextConsumerTest > testCommitMetadata PASSED

kafka.api.PlaintextConsumerTest > testRoundRobinAssignment STARTED

kafka.api.PlaintextConsumerTest > testRoundRobinAssignment PASSED

kafka.api.PlaintextConsumerTest > testPatternSubscription STARTED

kafka.api.PlaintextConsumerTest > testPatternSubscription PASSED

kafka.api.PlaintextConsumerTest > testCoordinatorFailover STARTED

kafka.api.PlaintextConsumerTest > testCoordinatorFailover PASSED

kafka.api.PlaintextConsumerTest > testSimpleConsumption STARTED

kafka.api.PlaintextConsumerTest > testSimpleConsumption PASSED

kafka.api.SaslPlainPlaintextConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslPlainPlaintextConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslPlainPlaintextConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslPlainPlaintextConsumerTest > testSimpleConsumption PASSED

kafka.api.ApiUtilsTest > testShortStringNonASCII STARTED

kafka.api.ApiUtilsTest > testShortStringNonASCII PASSED

kafka.api.ApiUtilsTest > testShortStringASCII STARTED

kafka.api.ApiUtilsTest > testShortStringASCII PASSED

kafka.api.SaslSslConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslSslConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslSslConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslSslConsumerTest > testSimpleConsumption PASSED

kafka.api.PlaintextProducerSendTest > testSerializerConstructors STARTED

kafka.api.PlaintextProducerSendTest > testSerializerConstructors PASSED

kafka.api.PlaintextProducerSendTest > 
testSendCompressedMessageWithLogAppendTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendCompressedMessageWithLogAppendTime PASSED

kafka.api.PlaintextProducerSendTest > testAutoCreateTopic STARTED

kafka.api.PlaintextProducerSendTest > testAutoCreateTopic PASSED

[GitHub] kafka pull request #2002: MINOR: Fix typos in documentation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2002


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2003: make documentation follow latest template

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2003


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2004: MINOR: Fix table of content in and cross reference...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/2004

MINOR: Fix table of content in and cross references in the documentation

Removed a non-existing reference in table of contents and fixed some 
section numbers.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka doc/fix_toc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2004.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2004


commit c9b44014c1ab91f7de01e2f3007e5954ce81a524
Author: Vahid Hashemian 
Date:   2016-10-10T21:43:43Z

MINOR: Fix table of content in and cross references in the documentation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2003: make documentation follow latest template

[derrickdoo]

GitHub user derrickdoo opened a pull request:

https://github.com/apache/kafka/pull/2003

make documentation follow latest template

Make the latest version of our docs follow the latest site template 
structure.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/derrickdoo/kafka docs-updates

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2003.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2003


commit a8f5cb5f0e9ce5232915a9a94072591a7561fe0b
Author: Derrick Or 
Date:   2016-10-10T21:16:58Z

make documentation follow latest template

add docs header




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2002: MINOR: Fix typos in documentation

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/2002

MINOR: Fix typos in documentation

And improve readability by adding proper punctuations.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka doc/fix_typos

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2002.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2002


commit a51642fa5e06c34705b3414d57ca218d4514e520
Author: Vahid Hashemian 
Date:   2016-10-10T21:10:37Z

MINOR: Fix typos in documentation

And improve readability by adding proper punctuations.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3175) topic not accessible after deletion even when delete.topic.enable is disabled

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy updated KAFKA-3175:
--
Resolution: Fixed
  Reviewer: Joel Koshy
Status: Resolved  (was: Patch Available)

> topic not accessible after deletion even when delete.topic.enable is disabled
> -
>
> Key: KAFKA-3175
> URL: https://issues.apache.org/jira/browse/KAFKA-3175
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Mayuresh Gharat
> Fix For: 0.10.1.1
>
>
> The can be reproduced with the following steps.
> 1. start ZK and 1 broker (with default delete.topic.enable=false)
> 2. create a topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test 
> --partition 1 --replication-factor 1
> 3. delete topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test
> 4. restart the broker
> Now topic test still shows up during topic description.
> bin/kafka-topics.sh --zookeeper localhost:2181 --describe
> Topic:testPartitionCount:1ReplicationFactor:1 Configs:
>   Topic: test Partition: 0Leader: 0   Replicas: 0 Isr: 0
> However, one can't produce to this topic any more.
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> [2016-01-29 17:55:24,527] WARN Error while fetching metadata with correlation 
> id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,725] WARN Error while fetching metadata with correlation 
> id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,828] WARN Error while fetching metadata with correlation 
> id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3175) topic not accessible after deletion even when delete.topic.enable is disabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563503#comment-15563503
 ] 

ASF GitHub Bot commented on KAFKA-3175:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/846


> topic not accessible after deletion even when delete.topic.enable is disabled
> -
>
> Key: KAFKA-3175
> URL: https://issues.apache.org/jira/browse/KAFKA-3175
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Mayuresh Gharat
> Fix For: 0.10.1.1
>
>
> The can be reproduced with the following steps.
> 1. start ZK and 1 broker (with default delete.topic.enable=false)
> 2. create a topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test 
> --partition 1 --replication-factor 1
> 3. delete topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test
> 4. restart the broker
> Now topic test still shows up during topic description.
> bin/kafka-topics.sh --zookeeper localhost:2181 --describe
> Topic:testPartitionCount:1ReplicationFactor:1 Configs:
>   Topic: test Partition: 0Leader: 0   Replicas: 0 Isr: 0
> However, one can't produce to this topic any more.
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> [2016-01-29 17:55:24,527] WARN Error while fetching metadata with correlation 
> id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,725] WARN Error while fetching metadata with correlation 
> id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,828] WARN Error while fetching metadata with correlation 
> id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #846: KAFKA-3175 : Topic not accessible after deletion ev...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/846


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : kafka-trunk-jdk7 #1616

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

[Renu Tewari]

Hi David
  This is a very timely KIP given the number of use cases in the streams
processing pipeline than need consumed log retention management.

Some questions that Becket and Dong asked just wanted to make sure are
described in the KIP.

1. How is the configuration setup per topic to know what is the set of
consumer groups that are "subscribed" to this topic whose committed offsets
will be tracked. Can we have more details on how this will be dynamically
tracked as consumers come and go.
2. Is there a timeout to determine if a consumer group has stopped
committing offsets to topic partitions that they had earlier consumed? Or
the consumed log retention will track each known consumer/consumers groups
committed offset and stop any cleaning if a consumer disappears after
consuming. This is to Dong's earlier question.
3. Can the log.retention value be set to 0 to indicate the log is set to be
cleaned to the min committed offset immediately after it has been consumed?

4. What guarantee is given on when the consumed log will eventually be
cleaned. If the log.retention timeout is enabled for a consumed offset and
a new consumer starts consuming from the beginning then is the min
committed offset value changed and the timer based on log.retention timeout
restarted?

 This kind of all relates to active and inactive consumers and if the set
changes dynamically how does the consumed log retention actually make
progress.

regards
renu


On Mon, Oct 10, 2016 at 1:05 AM, Dong Lin  wrote:

> Hey David,
>
> Thanks for reply. Please see comment inline.
>
> On Mon, Oct 10, 2016 at 12:40 AM, Pengwei (L) 
> wrote:
>
> > Hi Dong
> >Thanks for the questions:
> >
> > 1.  Now we don't distinguish inactive or active groups. Because in some
> > case maybe inactive group will become active again, and using the
> previous
> > commit offset.
> >
> > So we will not delete the log segment in the consumer retention if there
> > are some groups consume but not commit, but the log segment can be
> delete by
> >  the force retention.
> >
>
> So in the example I provided, the consumed log retention will be
> effectively disabled, right? This seems to be a real problem in operation
> -- we don't want log retention to be un-intentionally disabled simply
> because someone start a tool to consume from that topic. Either this KIP
> should provide a way to handle this, or there should be a way for operator
> to be aware of such case and be able to re-eanble consumed log retention
> for the topic. What do you think?
>
>
>
> > 2.  These configs are used to determine the out of date time of the
> > consumed retention, like the parameters of the force retention
> > (log.retention.hours, log.retention.minutes, log.retention.ms). For
> > example, users want the save the log for 3 days, after 3 days, kafka will
> > delete the log segments which are
> >
> > consumed by all the consumer group.  The log retention thread need these
> > parameters.
> >
> > It makes sense to have configs such as log.retention.ms -- it is used to
> make data available for up to a configured amount of time before it is
> deleted. My question is what is the use-case for making log available for
> another e.g. 3 days after it has been consumed by all consumer groups. The
> purpose of this KIP is to allow log to be deleted right as long as all
> interested consumer groups have consumed it. Can you provide a use-case for
> keeping log available for longer time after it has been consumed by all
> groups?
>
>
> >
> > Thanks,
> > David
> >
> >
> > > Hey David,
> > >
> > > Thanks for the KIP. Can you help with the following two questions:
> > >
> > > 1) If someone start a consumer (e.g. kafka-console-consumer) to
> consume a
> > > topic for debug/validation purpose, a randome consumer group may be
> > created
> > > and offset may be committed for this consumer group. If no offset
> commit
> > is
> > > made for this consumer group in the future, will this effectively
> > > disable consumed log retention for this topic? In other words, how do
> > this
> > > KIP distinguish active consumer group from inactive ones?
> > >
> > > 2) Why do we need new configs such as log.retention.commitoffset.
> hours?
> > Can
> > >we simply delete log segments if consumed log retention is enabled for
> > this
> > > topic and all consumer groups have consumed messages in the log
> segment?
> > >
> > > Thanks,
> > > Dong
> > >
> > >
> > >
> > >On Sat, Oct 8, 2016 at 2:15 AM, Pengwei (L) 
> > wrote:
> > >
> > > > Hi Becket,
> > > >
> > > >   Thanks for the feedback:
> > > > 1.  We use the simple consumer api to query the commit offset, so we
> > don't
> > > > need to specify the consumer group.
> > > > 2.  Every broker using the simple consumer api(OffsetFetchKey) to
> query
> > > > the commit offset in the log retention process.  The client can
> commit
> > > > offset or not.
> > > > 3.  It does not need to distinguish the 

[jira] [Resolved] (KAFKA-4244) Update our website look & feel

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-4244.

   Resolution: Fixed
Fix Version/s: 0.10.1.0

Issue resolved by pull request 1966
[https://github.com/apache/kafka/pull/1966]

> Update our website look & feel
> --
>
> Key: KAFKA-4244
> URL: https://issues.apache.org/jira/browse/KAFKA-4244
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
> Fix For: 0.10.1.0
>
>
> Our website deserves a facelift.
> This will be multi-part change:
> 1. Changes to the web pages in our normal GitHub to new headers, fix some 
> missing tags, etc.
> 2. Changes to the auto-get code to get protocol.html correct too
> 3. Deploy changes to website + update the header/footer/CSS in the website to 
> actual cause facelift.
> Please do not deploy changes to the website from our GitHub after #1 is done 
> but before #3 is complete. Hopefully, I'll be all done by Monday.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4244) Update our website look & feel

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563436#comment-15563436
 ] 

ASF GitHub Bot commented on KAFKA-4244:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1966


> Update our website look & feel
> --
>
> Key: KAFKA-4244
> URL: https://issues.apache.org/jira/browse/KAFKA-4244
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
>
> Our website deserves a facelift.
> This will be multi-part change:
> 1. Changes to the web pages in our normal GitHub to new headers, fix some 
> missing tags, etc.
> 2. Changes to the auto-get code to get protocol.html correct too
> 3. Deploy changes to website + update the header/footer/CSS in the website to 
> actual cause facelift.
> Please do not deploy changes to the website from our GitHub after #1 is done 
> but before #3 is complete. Hopefully, I'll be all done by Monday.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1966: KAFKA-4244: fixing formating issues in docs. missi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1966


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-4287) auto generate DynamicConfig in docs

[Jun Rao (JIRA)]

Jun Rao created KAFKA-4287:
--

 Summary: auto generate DynamicConfig in docs
 Key: KAFKA-4287
 URL: https://issues.apache.org/jira/browse/KAFKA-4287
 Project: Kafka
  Issue Type: Improvement
  Components: documentation
Affects Versions: 0.10.0.0
Reporter: Jun Rao


We now have dynamic configurations for users/clients/brokers. It would be 
useful to automatically generate the configuration docs like what we do for 
LogConfig,



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #960

[Apache Jenkins Server]

See 

Changes:

[jason] MINOR: Fixed incomplete sentence in introduction docs

--
[...truncated 13973 lines...]

org.apache.kafka.streams.state.internals.NamedCacheTest > shouldKeepTrackOfSize 
PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldReturnValueIfExists STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldReturnValueIfExists PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldNotGetValuesFromOtherStores STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldNotGetValuesFromOtherStores PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldGetApproximateEntriesAcrossAllStores STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldGetApproximateEntriesAcrossAllStores PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldReturnLongMaxValueOnOverflow STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldReturnLongMaxValueOnOverflow PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldReturnNullIfKeyDoesntExist STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldReturnNullIfKeyDoesntExist PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldThrowInvalidStoreExceptionDuringRebalance STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldThrowInvalidStoreExceptionDuringRebalance PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldThrowInvalidStoreExceptionOnAllDuringRebalance STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldThrowInvalidStoreExceptionOnAllDuringRebalance PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldSupportRange STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldSupportRange PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldFindValueForKeyWhenMultiStores STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldFindValueForKeyWhenMultiStores PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldThrowInvalidStoreExceptionOnRangeDuringRebalance STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldThrowInvalidStoreExceptionOnRangeDuringRebalance PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldSupportRangeAcrossMultipleKVStores STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldSupportRangeAcrossMultipleKVStores PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldSupportAllAcrossMultipleStores STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldSupportAllAcrossMultipleStores PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueStoreIteratorTest
 > shouldIterateOverRange STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueStoreIteratorTest
 > shouldIterateOverRange PASSED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldThrowInvalidStoreExceptionIfNoStoreOfTypeFound STARTED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldThrowInvalidStoreExceptionIfNoStoreOfTypeFound PASSED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldFindWindowStores STARTED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldFindWindowStores PASSED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldFindKeyValueStores STARTED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldFindKeyValueStores PASSED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldPutFetchFromCache STARTED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldPutFetchFromCache PASSED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldIterateCacheAndStore STARTED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldIterateCacheAndStore PASSED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldTakeValueFromCacheIfSameTimestampFlushedToRocks STARTED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldTakeValueFromCacheIfSameTimestampFlushedToRocks PASSED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 

[VOTE] 0.10.1.0 RC1

[Jason Gustafson]

Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 0.10.1.0. This is
a minor release that includes great new features including throttled
replication, secure quotas, time-based log searching, and queryable state
for Kafka Streams. A full list of the content can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.1.

One quick note on the docs. Because of all the recent improvements, the
documentation is still a bit out of sync with what's visible on the Kafka
homepage. This should be fixed soon (definitely before the release is
finalized).

Release notes for the 0.10.1.0 release:
http://home.apache.org/~jgus/kafka-0.10.1.0-rc1/RELEASE_NOTES.html


*** Please download, test and vote by Thursday, Oct 13, 1pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~jgus/kafka-0.10.1.0-rc1/


* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~jgus/kafka-0.10.1.0-rc1/javadoc/


* Tag to be voted upon (off 0.10.1 branch) is the 0.10.1.0-rc1 tag:
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=6eda15a97ffe17d636c390c0e0b28c8349993941

* Documentation:
http://kafka.apache.org/0101/documentation.html

* Protocol:
http://kafka.apache.org/0101/protocol.html

* Tests:
Unit tests: https://builds.apache.org/job/kafka-0.10.1-jdk7/59/
System tests:
http://testing.confluent.io/confluent-kafka-0-10-1-system-test-results/?prefix=2016-10-10--001.1476110532--apache--0.10.1--e696f17/

Thanks,

Jason

Build failed in Jenkins: kafka-trunk-jdk7 #1615

[Apache Jenkins Server]

See 

Changes:

[jason] MINOR: Fixed incomplete sentence in introduction docs

--
[...truncated 14063 lines...]
org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testStickiness PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexingTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexingTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingStatefulTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingStatefulTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleMultiSourceTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleMultiSourceTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testTopologyMetadata STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testTopologyMetadata PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexByNameTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexByNameTopology PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
shouldThrowStreamsExceptionAfterMaxAttempts STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
shouldThrowStreamsExceptionAfterMaxAttempts PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
shouldRetryWhenTimeoutExceptionOccursOnSend STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
shouldRetryWhenTimeoutExceptionOccursOnSend PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner PASSED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval STARTED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval PASSED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowProcessorStateExceptionOnInitializeOffsetsWhenAuthorizationException 
STARTED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowProcessorStateExceptionOnInitializeOffsetsWhenAuthorizationException 
PASSED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowProcessorStateExceptionOnInitializeOffsetsWhenKafkaException STARTED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowProcessorStateExceptionOnInitializeOffsetsWhenKafkaException PASSED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowWakeupExceptionOnInitializeOffsetsWhenWakeupException STARTED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowWakeupExceptionOnInitializeOffsetsWhenWakeupException PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldAddUserDefinedEndPointToSubscription STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldAddUserDefinedEndPointToSubscription PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithStandbyReplicas STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 

Jenkins build is back to normal : kafka-0.10.1-jdk7 #59

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (KAFKA-3514) Stream timestamp computation needs some further thoughts

[David J. Garcia (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562785#comment-15562785
 ] 

David J. Garcia edited comment on KAFKA-3514 at 10/10/16 6:42 PM:
--

maybe just use Max(all_paritions_ts) instead of min (for the punctuate logic)?


was (Author: djchooy):
maybe just use Max(all_paritions_ts) instead of min?

> Stream timestamp computation needs some further thoughts
> 
>
> Key: KAFKA-3514
> URL: https://issues.apache.org/jira/browse/KAFKA-3514
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>  Labels: architecture
> Fix For: 0.10.2.0
>
>
> Our current stream task's timestamp is used for punctuate function as well as 
> selecting which stream to process next (i.e. best effort stream 
> synchronization). And it is defined as the smallest timestamp over all 
> partitions in the task's partition group. This results in two unintuitive 
> corner cases:
> 1) observing a late arrived record would keep that stream's timestamp low for 
> a period of time, and hence keep being process until that late record. For 
> example take two partitions within the same task annotated by their 
> timestamps:
> {code}
> Stream A: 5, 6, 7, 8, 9, 1, 10
> {code}
> {code}
> Stream B: 2, 3, 4, 5
> {code}
> The late arrived record with timestamp "1" will cause stream A to be selected 
> continuously in the thread loop, i.e. messages with timestamp 5, 6, 7, 8, 9 
> until the record itself is dequeued and processed, then stream B will be 
> selected starting with timestamp 2.
> 2) an empty buffered partition will cause its timestamp to be not advanced, 
> and hence the task timestamp as well since it is the smallest among all 
> partitions. This may not be a severe problem compared with 1) above though.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

[Rajini Sivaram]

Gwen,

Thank you, will raise a separate KIP for a pluggable interface.

On Mon, Oct 10, 2016 at 5:55 PM, Gwen Shapira  wrote:

> I think it is fine to break the password store to an interface in a
> separate KIP. I actually love the idea of smaller KIPs dealing with
> more specific functionality. I just wasn't clear why it was rejected.
>
> Thank you for clarifying. I'm happy with current proposal.
>
> Gwen
>
> On Mon, Oct 10, 2016 at 2:17 AM, Rajini Sivaram
>  wrote:
> > Gwen,
> >
> > Thank you for reviewing the KIP.
> >
> > There has been interest in making the password verification in SASL/PLAIN
> > more pluggable. So I think it makes sense to have a pluggable interface
> > that can be adopted for any SASL mechanism rather than just SCRAM. With
> the
> > current proposal, you can plugin another Scram SaslServer implementation
> > with a different password store. This is similar to the current
> SASL/PLAIN
> > implementation.
> >
> > I agree that it will be good to make password stores more pluggable
> rather
> > than require users to override the whole SaslServer. I was going to look
> > into this later, but I can do it as part of this KIP. Will update the KIP
> > with a pluggable interface.
> >
> > Thank you,
> >
> > Rajini
> >
> >
> > On Fri, Oct 7, 2016 at 11:37 PM, Gwen Shapira  wrote:
> >
> >> Can you talk more about rejecting the option of making the password
> >> store pluggable? I am a bit uncomfortable with making ZK the one and
> >> only password store...
> >>
> >> On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram
> >>  wrote:
> >> > Hi all,
> >> >
> >> > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> >> > mechanisms to Kafka:
> >> >
> >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 84%3A+Support+SASL+SCRAM+mechanisms
> >> >
> >> >
> >> > Comments and suggestions are welcome.
> >> >
> >> > Thank you...
> >> >
> >> > Regards,
> >> >
> >> > Rajini
> >>
> >>
> >>
> >> --
> >> Gwen Shapira
> >> Product Manager | Confluent
> >> 650.450.2760 | @gwenshap
> >> Follow us: Twitter | blog
> >>
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>



-- 
Regards,

Rajini

[jira] [Commented] (KAFKA-4280) Add REST resource for showing available connector plugin configs

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562889#comment-15562889
 ] 

Ewen Cheslack-Postava commented on KAFKA-4280:
--

[~gwenshap] You can get the list of configs by submitting an empty config to 
the validation endpoint.

> Add REST resource for showing available connector plugin configs
> 
>
> Key: KAFKA-4280
> URL: https://issues.apache.org/jira/browse/KAFKA-4280
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Gwen Shapira
>Assignee: Ewen Cheslack-Postava
>
> Connector-plugins allow listing the plugs and validating configs, but we have 
> nothing (I think?) for listing available configuration properties.
> If this doesn't exist, would be good for usability to add it. If it does 
> exist, perhaps document it?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4286) metric reporter may hit NullPointerException during shutdown

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562886#comment-15562886
 ] 

Jun Rao commented on KAFKA-4286:


There are a few ways that we could fix this. One way is to obtain the 
KafkaMetric for "io-wait-ratio" once and cache it since this metric is only 
created once. This is also probably more efficient since we don't have to 
recompute the same metric name each time.

> metric reporter may hit NullPointerException during shutdown
> 
>
> Key: KAFKA-4286
> URL: https://issues.apache.org/jira/browse/KAFKA-4286
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>
> When we shut down a broker, our metric reporter could throw the following 
> exception.
> java.lang.NullPointerException
>   at kafka.network.Processor$$anon$2.value(SocketServer.scala:392)
>   at kafka.network.Processor$$anon$2.value(SocketServer.scala:390)
> This is because we report Yammer metric like the following and we de-register 
> the underlying Kafka metric when shutting down the socket server.
>   newGauge("IdlePercent",
> new Gauge[Double] {
>   def value = {
> metrics.metrics().get(metrics.metricName("io-wait-ratio", 
> "socket-server-metrics", metricTags)).value()
>   }
> },
> metricTags.asScala
>   )



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4286) metric reporter may hit NullPointerException during shutdown

[Jun Rao (JIRA)]

Jun Rao created KAFKA-4286:
--

 Summary: metric reporter may hit NullPointerException during 
shutdown
 Key: KAFKA-4286
 URL: https://issues.apache.org/jira/browse/KAFKA-4286
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.10.0.0
Reporter: Jun Rao


When we shut down a broker, our metric reporter could throw the following 
exception.
java.lang.NullPointerException
at kafka.network.Processor$$anon$2.value(SocketServer.scala:392)
at kafka.network.Processor$$anon$2.value(SocketServer.scala:390)

This is because we report Yammer metric like the following and we de-register 
the underlying Kafka metric when shutting down the socket server.
  newGauge("IdlePercent",
new Gauge[Double] {
  def value = {
metrics.metrics().get(metrics.metricName("io-wait-ratio", 
"socket-server-metrics", metricTags)).value()
  }
},
metricTags.asScala
  )




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Store flushing on commit.interval.ms from KIP-63 introduces aggregation latency

[Greg Fodor]

Hey Eno, thanks for the suggestion -- understood that my patch is not
something that could be accepted given the API change, I posted it to help
make the discussion concrete and because i needed a workaround. (Likely
we'll maintain this patch internally so we can move forward with the new
version, since the consumer heartbeat issue is something we really need
addressed.)

Looking at the code, it seems that setting the cache size to zero will
disable all caching. However, the previous version of Kafka Streams had a
local cache within the RocksDBStore to reduce I/O. If we were to set the
cache size to zero, my guess is we'd see a large increase in I/O relative
to the previous version since we would no longer have caching of any kind
even intra-store. By the looks of it there isn't an easy way to replicate
the same caching behavior as the old version of Kafka Streams in the new
system without increasing latency, but maybe I'm missing something.


On Oct 10, 2016 3:10 AM, "Eno Thereska"  wrote:

> Hi Greg,
>
> Thanks for trying 0.10.1. The best option you have for your specific app
> is to simply turn off caching by setting the cache size to 0. That should
> give you the old behaviour:
> streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG,
> 0L);
>
> Your PR is an alternative, but it requires changing the APIs and would
> require a KIP.
>
> Thanks
> Eno
>
> > On 9 Oct 2016, at 23:49, Greg Fodor  wrote:
> >
> > JIRA opened here: https://issues.apache.org/jira/browse/KAFKA-4281
> >
> > On Sun, Oct 9, 2016 at 2:02 AM, Greg Fodor  wrote:
> >> I went ahead and did some more testing, and it feels to me one option
> >> for resolving this issue is having a method on KGroupedStream which
> >> can be used to configure if the operations on it (reduce/aggregate)
> >> will forward immediately or not. I did a quick patch and was able to
> >> determine that if the records are forwarded immediately it resolves
> >> the issue I am seeing. Having it be done on a per-KGroupedStream basis
> >> would provide maximum flexibility.
> >>
> >> On Sun, Oct 9, 2016 at 1:06 AM, Greg Fodor  wrote:
> >>> I'm taking 0.10.1 for a spin on our existing Kafka Streams jobs and
> >>> I'm hitting what seems to be a serious issue (at least, for us) with
> >>> the changes brought about in KIP-63. In our job, we have a number of
> >>> steps in the topology where we perform a repartition and aggregation
> >>> on topics that require low latency. These topics have a very low
> >>> message volume but require subsecond latency for the aggregations to
> >>> complete since they are configuration data that drive the rest of the
> >>> job and need to be applied immediately.
> >>>
> >>> In 0.10.0, we performed a through (for repartitioning) and aggregateBy
> >>> and this resulted in minimal latency as the aggregateBy would just
> >>> result in a consumer attached to the output of the through and the
> >>> processor would consume + aggregate messages immediately passing them
> >>> to the next step in the topology.
> >>>
> >>> However, in 0.10.1 the aggregateBy API is no longer available and it
> >>> is necessary to pivot the data through a groupByKey and then
> >>> aggregate(). The problem is that this mechanism results in the
> >>> intermediate KTable state store storing the data as usual, but the
> >>> data is not forwarded downstream until the next store flush. (Due to
> >>> the use of ForwardingCacheFlushListener instead of calling forward()
> >>> during the process of the record.)
> >>>
> >>> As noted in KIP-63 and as I saw in the code, the flush interval of
> >>> state stores is commit.interval.ms. For us, this has been tuned to a
> >>> few seconds, and since we have a number of these aggregations in our
> >>> job sequentially, this now results in many seconds of latency in the
> >>> worst case for a tuple to travel through our topology.
> >>>
> >>> It seems too inflexible to have the flush interval always be the same
> >>> as the commit interval across all aggregates. For certain aggregations
> >>> which are idempotent regardless of messages being reprocessed, being
> >>> able to flush more often than the commit interval seems like a very
> >>> important option when lower latency is required. It would still make
> >>> sense to flush every commit as well, but having an additional
> >>> configuration to set the maximum time between state store flushes
> >>> seems like it would solve our problem.
> >>>
> >>> In our case, we'd set our flush interval to a few hundred ms. Ideally,
> >>> we would really prefer to be able to disable interval based flushing
> >>> altogether (and just put + forward all processed records) for certain
> >>> KTables that are low volume, latency sensitive, and which are
> >>> idempotent under message reprocessing.
> >>>
> >>> Thanks for any help! Right now the only option it seems is for us to
> >>> radically lower the commit interval and 

[GitHub] kafka pull request #1996: MINOR: Fixed introduction doc - wrong streams api ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1996


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3808) Transient failure in ReplicaVerificationToolTest

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562803#comment-15562803
 ] 

Ewen Cheslack-Postava commented on KAFKA-3808:
--

Seen again running against trunk commit 44d18d2:
Module: kafkatest.tests.tools.replica_verification_test
Class: ReplicaVerificationToolTest
Method: test_replica_lags
Test run report: 
http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-10-09--001.1476047058--apache--trunk--44d18d2/report.html
Archive with test run info: 
http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-10-09--001.1476047058--apache--trunk--44d18d2/ReplicaVerificationToolTest/test_replica_lags.tgz
The following is the exception that causes the failure:

test_id: 
2016-10-09--001.kafkatest.tests.tools.replica_verification_test.ReplicaVerificationToolTest.test_replica_lags
status: FAIL
run time: 1 minute 12.266 seconds
Timed out waiting to reach non-zero number of replica lags.
Traceback (most recent call last):
File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
 line 106, in run_all_tests
data = self.run_single_test()
File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
 line 162, in run_single_test
return self.current_test_context.function(self.current_test)
File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/tools/replica_verification_test.py",
 line 88, in test_replica_lags
err_msg="Timed out waiting to reach non-zero number of replica lags.")
File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/utils/util.py",
 line 36, in wait_until
raise TimeoutError(err_msg)
TimeoutError: Timed out waiting to reach non-zero number of replica lags.
I am not seeing anything in the recent commit history that looks related, so 
this may be an existing issue with the test. Nothing else obviously wrong 
popped out at me in the logs, but I don't know the details of this test.

> Transient failure in ReplicaVerificationToolTest
> 
>
> Key: KAFKA-3808
> URL: https://issues.apache.org/jira/browse/KAFKA-3808
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Reporter: Geoff Anderson
>
> {code}
> test_id:
> 2016-05-29--001.kafkatest.tests.tools.replica_verification_test.ReplicaVerificationToolTest.test_replica_lags
> status: FAIL
> run time:   1 minute 9.231 seconds
> Timed out waiting to reach non-zero number of replica lags.
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/tools/replica_verification_test.py",
>  line 88, in test_replica_lags
> err_msg="Timed out waiting to reach non-zero number of replica lags.")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Timed out waiting to reach non-zero number of replica lags.
> {code}
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-05-29--001.1464540508--apache--trunk--404b696/report.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4285) ReplicaVerificationToolTest.test_replica_lags: Timed out waiting to reach non-zero number of replica lags.

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-4285.
--
Resolution: Duplicate

> ReplicaVerificationToolTest.test_replica_lags: Timed out waiting to reach 
> non-zero number of replica lags.
> --
>
> Key: KAFKA-4285
> URL: https://issues.apache.org/jira/browse/KAFKA-4285
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ewen Cheslack-Postava
>Assignee: Flavio Junqueira
>  Labels: system-test-failure
>
> This failure happened on trunk against commit 44d18d2:
> Module: kafkatest.tests.tools.replica_verification_test
> Class:  ReplicaVerificationToolTest
> Method: test_replica_lags
> Test run report: 
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-10-09--001.1476047058--apache--trunk--44d18d2/report.html
> Archive with test run info: 
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-10-09--001.1476047058--apache--trunk--44d18d2/ReplicaVerificationToolTest/test_replica_lags.tgz
> The following is the exception that causes the failure:
> {quote}
> 
> test_id:
> 2016-10-09--001.kafkatest.tests.tools.replica_verification_test.ReplicaVerificationToolTest.test_replica_lags
> status: FAIL
> run time:   1 minute 12.266 seconds
> Timed out waiting to reach non-zero number of replica lags.
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/tools/replica_verification_test.py",
>  line 88, in test_replica_lags
> err_msg="Timed out waiting to reach non-zero number of replica lags.")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Timed out waiting to reach non-zero number of replica lags.
> {quote}
> I am not seeing anything in the recent commit history that looks related, so 
> this may be an existing issue with the test. Nothing else obviously wrong 
> popped out at me in the logs, but I don't know the details of this test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

[Gwen Shapira]

I think it is fine to break the password store to an interface in a
separate KIP. I actually love the idea of smaller KIPs dealing with
more specific functionality. I just wasn't clear why it was rejected.

Thank you for clarifying. I'm happy with current proposal.

Gwen

On Mon, Oct 10, 2016 at 2:17 AM, Rajini Sivaram
 wrote:
> Gwen,
>
> Thank you for reviewing the KIP.
>
> There has been interest in making the password verification in SASL/PLAIN
> more pluggable. So I think it makes sense to have a pluggable interface
> that can be adopted for any SASL mechanism rather than just SCRAM. With the
> current proposal, you can plugin another Scram SaslServer implementation
> with a different password store. This is similar to the current SASL/PLAIN
> implementation.
>
> I agree that it will be good to make password stores more pluggable rather
> than require users to override the whole SaslServer. I was going to look
> into this later, but I can do it as part of this KIP. Will update the KIP
> with a pluggable interface.
>
> Thank you,
>
> Rajini
>
>
> On Fri, Oct 7, 2016 at 11:37 PM, Gwen Shapira  wrote:
>
>> Can you talk more about rejecting the option of making the password
>> store pluggable? I am a bit uncomfortable with making ZK the one and
>> only password store...
>>
>> On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram
>>  wrote:
>> > Hi all,
>> >
>> > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
>> > mechanisms to Kafka:
>> >
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 84%3A+Support+SASL+SCRAM+mechanisms
>> >
>> >
>> > Comments and suggestions are welcome.
>> >
>> > Thank you...
>> >
>> > Regards,
>> >
>> > Rajini
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>
>
>
>
> --
> Regards,
>
> Rajini



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[jira] [Created] (KAFKA-4285) ReplicaVerificationToolTest.test_replica_lags: Timed out waiting to reach non-zero number of replica lags.

[Ewen Cheslack-Postava (JIRA)]

Ewen Cheslack-Postava created KAFKA-4285:


 Summary: ReplicaVerificationToolTest.test_replica_lags: Timed out 
waiting to reach non-zero number of replica lags.
 Key: KAFKA-4285
 URL: https://issues.apache.org/jira/browse/KAFKA-4285
 Project: Kafka
  Issue Type: Bug
Reporter: Ewen Cheslack-Postava
Assignee: Flavio Junqueira


This failure happened on trunk against commit 44d18d2:

Module: kafkatest.tests.tools.replica_verification_test
Class:  ReplicaVerificationToolTest
Method: test_replica_lags

Test run report: 
http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-10-09--001.1476047058--apache--trunk--44d18d2/report.html

Archive with test run info: 
http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-10-09--001.1476047058--apache--trunk--44d18d2/ReplicaVerificationToolTest/test_replica_lags.tgz

The following is the exception that causes the failure:
{quote}

test_id:
2016-10-09--001.kafkatest.tests.tools.replica_verification_test.ReplicaVerificationToolTest.test_replica_lags
status: FAIL
run time:   1 minute 12.266 seconds


Timed out waiting to reach non-zero number of replica lags.
Traceback (most recent call last):
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
 line 106, in run_all_tests
data = self.run_single_test()
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
 line 162, in run_single_test
return self.current_test_context.function(self.current_test)
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/tools/replica_verification_test.py",
 line 88, in test_replica_lags
err_msg="Timed out waiting to reach non-zero number of replica lags.")
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/utils/util.py",
 line 36, in wait_until
raise TimeoutError(err_msg)
TimeoutError: Timed out waiting to reach non-zero number of replica lags.
{quote}

I am not seeing anything in the recent commit history that looks related, so 
this may be an existing issue with the test. Nothing else obviously wrong 
popped out at me in the logs, but I don't know the details of this test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3514) Stream timestamp computation needs some further thoughts

[David J. Garcia (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562785#comment-15562785
 ] 

David J. Garcia commented on KAFKA-3514:


maybe just use Max(all_paritions_ts) instead of min?

> Stream timestamp computation needs some further thoughts
> 
>
> Key: KAFKA-3514
> URL: https://issues.apache.org/jira/browse/KAFKA-3514
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>  Labels: architecture
> Fix For: 0.10.2.0
>
>
> Our current stream task's timestamp is used for punctuate function as well as 
> selecting which stream to process next (i.e. best effort stream 
> synchronization). And it is defined as the smallest timestamp over all 
> partitions in the task's partition group. This results in two unintuitive 
> corner cases:
> 1) observing a late arrived record would keep that stream's timestamp low for 
> a period of time, and hence keep being process until that late record. For 
> example take two partitions within the same task annotated by their 
> timestamps:
> {code}
> Stream A: 5, 6, 7, 8, 9, 1, 10
> {code}
> {code}
> Stream B: 2, 3, 4, 5
> {code}
> The late arrived record with timestamp "1" will cause stream A to be selected 
> continuously in the thread loop, i.e. messages with timestamp 5, 6, 7, 8, 9 
> until the record itself is dequeued and processed, then stream B will be 
> selected starting with timestamp 2.
> 2) an empty buffered partition will cause its timestamp to be not advanced, 
> and hence the task timestamp as well since it is the smallest among all 
> partitions. This may not be a severe problem compared with 1) above though.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1994: MINOR: Introduction Doc: Fixed incomplete sentence

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1994


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-81: Max in-flight fetches

[Ismael Juma]

Hi Mickael,

Thanks for the KIP. A quick comment on the rejected alternative of using a
bounded memory pool:

"While this might be the best way to handle that on the server side it's
unclear if this would suit the client well. Usually the client has a rough
idea about how many partitions it will be subscribed to so it's easier to
size the maximum number of concurrent fetches."

I think this should be discussed in more detail. The producer (a client)
uses a `BufferPool`, so we should also explain why the consumer should
follow a different approach. Also, with KIP-74, the number of partitions is
less relevant than the number of brokers with partitions that a consumer is
subscribed to (which can change as the Kafka cluster size changes).

I think it's also worth separating implementation from the config options.
For example, one could configure a memory limit with an implementation that
limits the number of concurrent fetches or uses a bounded memory pool. Are
there other good reasons to have an explicit concurrent fetches limit per
consumer config? If so, it would be good to mention them in the KIP.

Ismael

On Mon, Oct 10, 2016 at 2:41 PM, Mickael Maison 
wrote:

> Hi all,
>
> I would like to discuss the following KIP proposal:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-81%3A+
> Max+in-flight+fetches
>
>
> Feedback and comments are welcome.
> Thanks !
>
> Mickael
>

[jira] [Updated] (KAFKA-4283) records deleted from CachingKeyValueStore still appear in range and all queries

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy updated KAFKA-4283:
--
Status: Patch Available  (was: In Progress)

> records deleted from CachingKeyValueStore still appear in range and all 
> queries
> ---
>
> Key: KAFKA-4283
> URL: https://issues.apache.org/jira/browse/KAFKA-4283
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.10.1.1
>
>
> Records deleted from CachingKeyValueStore appear in range and all queries. 
> The deleted record is replaced with an LRUCacheEntry(null), so that when a 
> flush happens the deletion can be sent downstream and removed from the 
> underlying store.
> As this is valid and needed, the iterator used when querying a cached store 
> needs to be aware of the entries where LRUCacheEntry.value == null and skip 
> over them



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2001: KAFKA-4283: records deleted from CachingKeyValueSt...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/2001

KAFKA-4283: records deleted from CachingKeyValueStore still appear in range 
and all queries

Records that are deleted/removed from the CachingKeyValueStore shouldn't 
appear in range and all queries.
Modified the iterator such that it skips over the deleted records.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-4283

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2001.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2001


commit e75f8c315b91a04a7b66ffa88a8bf83b973cb99c
Author: Damian Guy 
Date:   2016-10-10T13:39:47Z

skip items that have been deleted from the cache in queries




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4283) records deleted from CachingKeyValueStore still appear in range and all queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562344#comment-15562344
 ] 

ASF GitHub Bot commented on KAFKA-4283:
---

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/2001

KAFKA-4283: records deleted from CachingKeyValueStore still appear in range 
and all queries

Records that are deleted/removed from the CachingKeyValueStore shouldn't 
appear in range and all queries.
Modified the iterator such that it skips over the deleted records.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-4283

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2001.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2001


commit e75f8c315b91a04a7b66ffa88a8bf83b973cb99c
Author: Damian Guy 
Date:   2016-10-10T13:39:47Z

skip items that have been deleted from the cache in queries




> records deleted from CachingKeyValueStore still appear in range and all 
> queries
> ---
>
> Key: KAFKA-4283
> URL: https://issues.apache.org/jira/browse/KAFKA-4283
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.10.1.1
>
>
> Records deleted from CachingKeyValueStore appear in range and all queries. 
> The deleted record is replaced with an LRUCacheEntry(null), so that when a 
> flush happens the deletion can be sent downstream and removed from the 
> underlying store.
> As this is valid and needed, the iterator used when querying a cached store 
> needs to be aware of the entries where LRUCacheEntry.value == null and skip 
> over them



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[DISCUSS] KIP-81: Max in-flight fetches

[Mickael Maison]

Hi all,

I would like to discuss the following KIP proposal:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-81%3A+Max+in-flight+fetches


Feedback and comments are welcome.
Thanks !

Mickael

[jira] [Updated] (KAFKA-4284) Partitioner never closed by producer

[Theo Hultberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Theo Hultberg updated KAFKA-4284:
-
Description: 
Partitioners are never closed by the producer, even though the Partitioner 
interface has a close method.

I looked at KAFKA-2091 and it seems like the close method has been there from 
the beginning, but never been used.

  was:
Partitioners are never closed by the producer, even though the Partitioner 
interface has a close method.

I looked at KAFKA-2091 and it seems like the close method has been there from 
the beginning, but never been used.

I've made a pull request with a fix, it can be found here: 
https://github.com/apache/kafka/pull/2000


> Partitioner never closed by producer
> 
>
> Key: KAFKA-4284
> URL: https://issues.apache.org/jira/browse/KAFKA-4284
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.10.0.1
>Reporter: Theo Hultberg
>Priority: Minor
>
> Partitioners are never closed by the producer, even though the Partitioner 
> interface has a close method.
> I looked at KAFKA-2091 and it seems like the close method has been there from 
> the beginning, but never been used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4284) Partitioner never closed by producer

[Theo Hultberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Theo Hultberg updated KAFKA-4284:
-
Status: Patch Available  (was: Open)

Patch available here: https://github.com/apache/kafka/pull/2000

> Partitioner never closed by producer
> 
>
> Key: KAFKA-4284
> URL: https://issues.apache.org/jira/browse/KAFKA-4284
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.10.0.1
>Reporter: Theo Hultberg
>Priority: Minor
>
> Partitioners are never closed by the producer, even though the Partitioner 
> interface has a close method.
> I looked at KAFKA-2091 and it seems like the close method has been there from 
> the beginning, but never been used.
> I've made a pull request with a fix, it can be found here: 
> https://github.com/apache/kafka/pull/2000



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4284) Partitioner never closed by producer

[Theo Hultberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Theo Hultberg updated KAFKA-4284:
-
Description: 
Partitioners are never closed by the producer, even though the Partitioner 
interface has a close method.

I looked at KAFKA-2091 and it seems like the close method has been there from 
the beginning, but never been used.

I've made a pull request with a fix, it can be found here: 
https://github.com/apache/kafka/pull/2000

  was:
Partitioners are never closed by the producer, even though the Partitioner 
interface has a close method.

I looked at KAFKA-2091 and it seems like the close method has been there from 
the beginning, but never been used.

There's 


> Partitioner never closed by producer
> 
>
> Key: KAFKA-4284
> URL: https://issues.apache.org/jira/browse/KAFKA-4284
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.10.0.1
>Reporter: Theo Hultberg
>Priority: Minor
>
> Partitioners are never closed by the producer, even though the Partitioner 
> interface has a close method.
> I looked at KAFKA-2091 and it seems like the close method has been there from 
> the beginning, but never been used.
> I've made a pull request with a fix, it can be found here: 
> https://github.com/apache/kafka/pull/2000



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4284) Partitioner never closed by producer

[Theo Hultberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Theo Hultberg updated KAFKA-4284:
-
Description: 
Partitioners are never closed by the producer, even though the Partitioner 
interface has a close method.

I looked at KAFKA-2091 and it seems like the close method has been there from 
the beginning, but never been used.

There's 

  was:
Partitioners are never closed by the producer, even though the Partitioner 
interface has a close method.

I looked at KAFKA-2091 and it seems like the close method has been there from 
the beginning, but never been used.

I will push up a pull request for this shortly.


> Partitioner never closed by producer
> 
>
> Key: KAFKA-4284
> URL: https://issues.apache.org/jira/browse/KAFKA-4284
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.10.0.1
>Reporter: Theo Hultberg
>Priority: Minor
>
> Partitioners are never closed by the producer, even though the Partitioner 
> interface has a close method.
> I looked at KAFKA-2091 and it seems like the close method has been there from 
> the beginning, but never been used.
> There's 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2000: Make Partitioner a Closeable and close it when clo...

[iconara]

GitHub user iconara opened a pull request:

https://github.com/apache/kafka/pull/2000

Make Partitioner a Closeable and close it when closing the producer

Even though Partitioner has a close method it is not closed when the 
producer is closed. Serializers, interceptors and metrics are all closed, so 
partitioners should be closed to.

Looking at [KAFKA-2091](https://issues.apache.org/jira/browse/KAFKA-2091) 
(d6c45c70fb9773043766446e88370db9709e7995) that introduced the `Partitioner` 
interface it looks like the intention was that the producer should close the 
partitioner.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iconara/kafka kafka-4284

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2000.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2000


commit dcf85e553f69df7a3327166c56e16f3a12694c6c
Author: Theo 
Date:   2016-10-10T11:06:12Z

Make Partitioner a Closeable and close it when closing the producer

Even though Partitioner has a close method it is not closed when the 
producer is closed. Serializers, interceptors and metrics are all closed, so 
partitioners should be closed to.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2091) Expose a Partitioner interface in the new producer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562055#comment-15562055
 ] 

ASF GitHub Bot commented on KAFKA-2091:
---

GitHub user iconara opened a pull request:

https://github.com/apache/kafka/pull/2000

Make Partitioner a Closeable and close it when closing the producer

Even though Partitioner has a close method it is not closed when the 
producer is closed. Serializers, interceptors and metrics are all closed, so 
partitioners should be closed to.

Looking at [KAFKA-2091](https://issues.apache.org/jira/browse/KAFKA-2091) 
(d6c45c70fb9773043766446e88370db9709e7995) that introduced the `Partitioner` 
interface it looks like the intention was that the producer should close the 
partitioner.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iconara/kafka kafka-4284

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2000.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2000


commit dcf85e553f69df7a3327166c56e16f3a12694c6c
Author: Theo 
Date:   2016-10-10T11:06:12Z

Make Partitioner a Closeable and close it when closing the producer

Even though Partitioner has a close method it is not closed when the 
producer is closed. Serializers, interceptors and metrics are all closed, so 
partitioners should be closed to.




> Expose a Partitioner interface in the new producer
> --
>
> Key: KAFKA-2091
> URL: https://issues.apache.org/jira/browse/KAFKA-2091
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jay Kreps
>Assignee: Sriharsha Chintalapani
> Attachments: KAFKA-2091.patch, KAFKA-2091_2015-05-27_15:50:18.patch
>
>
> In the new producer you can pass in a key or hard code the partition as part 
> of ProducerRecord.
> Internally we are using a class
> {code}
> class Partitioner {
> public int partition(String topic, byte[] key, Integer partition, Cluster 
> cluster) {...}
> }
> {code}
> This class uses the specified partition if there is one; uses a hash of the 
> key if there isn't a partition but there is a key; and simply chooses a 
> partition round robin if there is neither a partition nor a key.
> However there are several partitioning strategies that could be useful that 
> we don't support out of the box. 
> An example would be having each producer periodically choose a random 
> partition. This tends to be the most efficient since all data goes to one 
> server and uses the fewest TCP connections, however it only produces good 
> load balancing if there are many producers.
> Of course a user can do this now by just setting the partition manually, but 
> that is a bit inconvenient if you need to do that across a bunch of apps 
> since each will need to remember to set the partition every time.
> The idea would be to expose a configuration to set the partitioner 
> implementation like
> {code}
> partitioner.class=org.apache.kafka.producer.DefaultPartitioner
> {code}
> This would default to the existing partitioner implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4284) Partitioner never closed by producer

[Theo Hultberg (JIRA)]

Theo Hultberg created KAFKA-4284:


 Summary: Partitioner never closed by producer
 Key: KAFKA-4284
 URL: https://issues.apache.org/jira/browse/KAFKA-4284
 Project: Kafka
  Issue Type: Bug
  Components: producer 
Affects Versions: 0.10.0.1
Reporter: Theo Hultberg
Priority: Minor


Partitioners are never closed by the producer, even though the Partitioner 
interface has a close method.

I looked at KAFKA-2091 and it seems like the close method has been there from 
the beginning, but never been used.

I will push up a pull request for this shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4283) records deleted from CachingKeyValueStore still appear in range and all queries

[Damian Guy (JIRA)]

Damian Guy created KAFKA-4283:
-

 Summary: records deleted from CachingKeyValueStore still appear in 
range and all queries
 Key: KAFKA-4283
 URL: https://issues.apache.org/jira/browse/KAFKA-4283
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.10.1.0
Reporter: Damian Guy
Assignee: Damian Guy
 Fix For: 0.10.1.1


Records deleted from CachingKeyValueStore appear in range and all queries. The 
deleted record is replaced with an LRUCacheEntry(null), so that when a flush 
happens the deletion can be sent downstream and removed from the underlying 
store.
As this is valid and needed, the iterator used when querying a cached store 
needs to be aware of the entries where LRUCacheEntry.value == null and skip 
over them



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-4283) records deleted from CachingKeyValueStore still appear in range and all queries

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-4283 started by Damian Guy.
-
> records deleted from CachingKeyValueStore still appear in range and all 
> queries
> ---
>
> Key: KAFKA-4283
> URL: https://issues.apache.org/jira/browse/KAFKA-4283
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.10.1.1
>
>
> Records deleted from CachingKeyValueStore appear in range and all queries. 
> The deleted record is replaced with an LRUCacheEntry(null), so that when a 
> flush happens the deletion can be sent downstream and removed from the 
> underlying store.
> As this is valid and needed, the iterator used when querying a cached store 
> needs to be aware of the entries where LRUCacheEntry.value == null and skip 
> over them



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1999: 2.11-0.10.0.1 can't see /consumers/group node

[cyfonly]

GitHub user cyfonly opened a pull request:

https://github.com/apache/kafka/pull/1999

2.11-0.10.0.1 can't see /consumers/group node

My producers and consumers all connected to kafka borkers and can exchange 
messages quite well, I can see "/consumers" node in zookeeper but there's 
nothing under "/consumers" node.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/kafka 0.10.1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1999.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1999


commit f66e88a30b2aabd02bc0ef35fa27f8bd9b35466f
Author: Ben Stopford 
Date:   2016-09-20T06:17:23Z

KAFKA-4193; Fix for intermittent failure in FetcherTest

Author: Ben Stopford 

Reviewers: Jason Gustafson 

Closes #1881 from benstopford/KAFKA-4193

(cherry picked from commit f396fdac197409fb955f00a6f642f04e4926ba41)
Signed-off-by: Jason Gustafson 

commit fc5f48aad4952df88147675a663ad034ce15d13d
Author: Eno Thereska 
Date:   2016-09-20T10:33:50Z

HOTFIX: Added check for metadata unavailable

Author: Eno Thereska 

Reviewers: Damian Guy , Ismael Juma 


Closes #1887 from enothereska/hotfix-metadata-unavailable

commit d48415f185d1882f0a3b89a3ce03ea84893393ba
Author: Ben Stopford 
Date:   2016-09-20T13:53:48Z

KAFKA-4184; Intermittent failures in 
ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle

Build is unstable, so it's hard to validate this change. Of the various 
builds up until 11am BST the test ran twice and passed twice.

Author: Ben Stopford 

Reviewers: Ismael Juma 

Closes #1873 from benstopford/KAFKA-4184

(cherry picked from commit 3663275cf066b7715cc11b26fd9c144bbff1c373)
Signed-off-by: Ismael Juma 

commit 29e30a79e5ba4f137d77943976b1e8e77f6ccaac
Author: Ben Stopford 
Date:   2016-09-20T14:41:14Z

KAFKA-4197; Make ReassignPartitionsTest System Test move data

The ReassignPartitionsTest system tests doesn't reassign any replicas (i.e. 
move data).

This is a simple issue. It uses a 3 node cluster with replication factor of 
3, so whilst the replicas are jumbled around, nothing actually is moved from 
machine to machine when the assignment is executed.

This fix just ups the number of nodes to 4 so things move.

Tests pass locally.
There are runs pending on the two branch builders

Passes:
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/551/
https://jenkins.confluent.io/job/system-test-kafka-branch-builder-2/94/
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/553/
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/554/
https://jenkins.confluent.io/job/system-test-kafka-branch-builder-2/95

Failures:
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/552 => 
_RuntimeError: There aren't enough available nodes to satisfy the resource 
request. Total cluster size: 1, Requested: 4, Already allocated: 1, Available: 
0._ Which I assume to do with the test env.

Author: Ben Stopford 

Reviewers: Ismael Juma 

Closes #1892 from benstopford/fix_reassignment_test

(cherry picked from commit 4f821830bc6b726cddf90999fff76006745b1a3f)
Signed-off-by: Ismael Juma 

commit 5b29bb8b039c96634d82352c7e17922c83dad48f
Author: Damian Guy 
Date:   2016-09-21T18:11:12Z

MINOR: add javadoc comment to PersistenKeyValueFactory.enableCaching

missing javadoc on public API method PersistenKeyValueFactory.enableCaching

Author: Damian Guy 

Reviewers: Eno Thereska, Guozhang Wang

Closes #1891 from dguy/minor-java-doc

(cherry picked from commit 24f81ea764a493b4422b6a3ef6b3e771d0e4d63b)
Signed-off-by: Guozhang Wang 

commit be20ea52892c91f59323e0be1108f689e5a44f95
Author: Damian Guy 
Date:   2016-09-21T18:13:39Z

MINOR: remove unused code from InternalTopicManager

Remove isValidCleanupPolicy and related fields as they are never used.

Author: Damian Guy 

Reviewers: Eno Thereska, Guozhang Wang

Closes #1888 from dguy/minor-remove-unused

(cherry picked from commit a632716a3c9a871f325c6f13aefa9aed0add4b82)
Signed-off-by: Guozhang Wang 

Re: Store flushing on commit.interval.ms from KIP-63 introduces aggregation latency

[Eno Thereska]

Hi Greg,

Thanks for trying 0.10.1. The best option you have for your specific app is to 
simply turn off caching by setting the cache size to 0. That should give you 
the old behaviour:
streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0L);

Your PR is an alternative, but it requires changing the APIs and would require 
a KIP. 

Thanks
Eno

> On 9 Oct 2016, at 23:49, Greg Fodor  wrote:
> 
> JIRA opened here: https://issues.apache.org/jira/browse/KAFKA-4281
> 
> On Sun, Oct 9, 2016 at 2:02 AM, Greg Fodor  wrote:
>> I went ahead and did some more testing, and it feels to me one option
>> for resolving this issue is having a method on KGroupedStream which
>> can be used to configure if the operations on it (reduce/aggregate)
>> will forward immediately or not. I did a quick patch and was able to
>> determine that if the records are forwarded immediately it resolves
>> the issue I am seeing. Having it be done on a per-KGroupedStream basis
>> would provide maximum flexibility.
>> 
>> On Sun, Oct 9, 2016 at 1:06 AM, Greg Fodor  wrote:
>>> I'm taking 0.10.1 for a spin on our existing Kafka Streams jobs and
>>> I'm hitting what seems to be a serious issue (at least, for us) with
>>> the changes brought about in KIP-63. In our job, we have a number of
>>> steps in the topology where we perform a repartition and aggregation
>>> on topics that require low latency. These topics have a very low
>>> message volume but require subsecond latency for the aggregations to
>>> complete since they are configuration data that drive the rest of the
>>> job and need to be applied immediately.
>>> 
>>> In 0.10.0, we performed a through (for repartitioning) and aggregateBy
>>> and this resulted in minimal latency as the aggregateBy would just
>>> result in a consumer attached to the output of the through and the
>>> processor would consume + aggregate messages immediately passing them
>>> to the next step in the topology.
>>> 
>>> However, in 0.10.1 the aggregateBy API is no longer available and it
>>> is necessary to pivot the data through a groupByKey and then
>>> aggregate(). The problem is that this mechanism results in the
>>> intermediate KTable state store storing the data as usual, but the
>>> data is not forwarded downstream until the next store flush. (Due to
>>> the use of ForwardingCacheFlushListener instead of calling forward()
>>> during the process of the record.)
>>> 
>>> As noted in KIP-63 and as I saw in the code, the flush interval of
>>> state stores is commit.interval.ms. For us, this has been tuned to a
>>> few seconds, and since we have a number of these aggregations in our
>>> job sequentially, this now results in many seconds of latency in the
>>> worst case for a tuple to travel through our topology.
>>> 
>>> It seems too inflexible to have the flush interval always be the same
>>> as the commit interval across all aggregates. For certain aggregations
>>> which are idempotent regardless of messages being reprocessed, being
>>> able to flush more often than the commit interval seems like a very
>>> important option when lower latency is required. It would still make
>>> sense to flush every commit as well, but having an additional
>>> configuration to set the maximum time between state store flushes
>>> seems like it would solve our problem.
>>> 
>>> In our case, we'd set our flush interval to a few hundred ms. Ideally,
>>> we would really prefer to be able to disable interval based flushing
>>> altogether (and just put + forward all processed records) for certain
>>> KTables that are low volume, latency sensitive, and which are
>>> idempotent under message reprocessing.
>>> 
>>> Thanks for any help! Right now the only option it seems is for us to
>>> radically lower the commit interval and accept any leftover latency,
>>> but unless we can find a sweet spot this may be a blocker for us to
>>> moving to 0.10.1.

[jira] [Work started] (KAFKA-4081) Consumer API consumer new interface commitSyn does not verify the validity of offset

[Mickael Maison (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-4081 started by Mickael Maison.
-
> Consumer API consumer new interface commitSyn does not verify the validity of 
> offset
> 
>
> Key: KAFKA-4081
> URL: https://issues.apache.org/jira/browse/KAFKA-4081
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.1
>Reporter: lifeng
>Assignee: Mickael Maison
>
> Consumer API consumer new interface commitSyn synchronization update offset, 
> for the illegal offset successful return, illegal offset<0 or offset>hw



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-4180) Shared authentification with multiple actives Kafka producers/consumers

[Mickael Maison (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-4180 started by Mickael Maison.
-
> Shared authentification with multiple actives Kafka producers/consumers
> ---
>
> Key: KAFKA-4180
> URL: https://issues.apache.org/jira/browse/KAFKA-4180
> Project: Kafka
>  Issue Type: Bug
>  Components: producer , security
>Affects Versions: 0.10.0.1
>Reporter: Guillaume Grossetie
>Assignee: Mickael Maison
>  Labels: authentication, jaas, loginmodule, plain, producer, 
> sasl, user
>
> I'm using Kafka 0.10.0.1 with an SASL authentication on the client:
> {code:title=kafka_client_jaas.conf|borderStyle=solid}
> KafkaClient {
> org.apache.kafka.common.security.plain.PlainLoginModule required
> username="guillaume"
> password="secret";
> };
> {code}
> When using multiple Kafka producers the authentification is shared [1]. In 
> other words it's not currently possible to have multiple Kafka producers in a 
> JVM process.
> Am I missing something ? How can I have multiple active Kafka producers with 
> different credentials ?
> My use case is that I have an application that send messages to multiples 
> clusters (one cluster for logs, one cluster for metrics, one cluster for 
> business data).
> [1] 
> https://github.com/apache/kafka/blob/69ebf6f7be2fc0e471ebd5b7a166468017ff2651/clients/src/main/java/org/apache/kafka/common/security/authenticator/LoginManager.java#L35



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

[Rajini Sivaram]

Gwen,

Thank you for reviewing the KIP.

There has been interest in making the password verification in SASL/PLAIN
more pluggable. So I think it makes sense to have a pluggable interface
that can be adopted for any SASL mechanism rather than just SCRAM. With the
current proposal, you can plugin another Scram SaslServer implementation
with a different password store. This is similar to the current SASL/PLAIN
implementation.

I agree that it will be good to make password stores more pluggable rather
than require users to override the whole SaslServer. I was going to look
into this later, but I can do it as part of this KIP. Will update the KIP
with a pluggable interface.

Thank you,

Rajini


On Fri, Oct 7, 2016 at 11:37 PM, Gwen Shapira  wrote:

> Can you talk more about rejecting the option of making the password
> store pluggable? I am a bit uncomfortable with making ZK the one and
> only password store...
>
> On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram
>  wrote:
> > Hi all,
> >
> > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> > mechanisms to Kafka:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 84%3A+Support+SASL+SCRAM+mechanisms
> >
> >
> > Comments and suggestions are welcome.
> >
> > Thank you...
> >
> > Regards,
> >
> > Rajini
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>



-- 
Regards,

Rajini

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

[Dong Lin]

Hey David,

Thanks for reply. Please see comment inline.

On Mon, Oct 10, 2016 at 12:40 AM, Pengwei (L)  wrote:

> Hi Dong
>Thanks for the questions:
>
> 1.  Now we don't distinguish inactive or active groups. Because in some
> case maybe inactive group will become active again, and using the previous
> commit offset.
>
> So we will not delete the log segment in the consumer retention if there
> are some groups consume but not commit, but the log segment can be delete by
>  the force retention.
>

So in the example I provided, the consumed log retention will be
effectively disabled, right? This seems to be a real problem in operation
-- we don't want log retention to be un-intentionally disabled simply
because someone start a tool to consume from that topic. Either this KIP
should provide a way to handle this, or there should be a way for operator
to be aware of such case and be able to re-eanble consumed log retention
for the topic. What do you think?



> 2.  These configs are used to determine the out of date time of the
> consumed retention, like the parameters of the force retention
> (log.retention.hours, log.retention.minutes, log.retention.ms). For
> example, users want the save the log for 3 days, after 3 days, kafka will
> delete the log segments which are
>
> consumed by all the consumer group.  The log retention thread need these
> parameters.
>
> It makes sense to have configs such as log.retention.ms -- it is used to
make data available for up to a configured amount of time before it is
deleted. My question is what is the use-case for making log available for
another e.g. 3 days after it has been consumed by all consumer groups. The
purpose of this KIP is to allow log to be deleted right as long as all
interested consumer groups have consumed it. Can you provide a use-case for
keeping log available for longer time after it has been consumed by all
groups?


>
> Thanks,
> David
>
>
> > Hey David,
> >
> > Thanks for the KIP. Can you help with the following two questions:
> >
> > 1) If someone start a consumer (e.g. kafka-console-consumer) to consume a
> > topic for debug/validation purpose, a randome consumer group may be
> created
> > and offset may be committed for this consumer group. If no offset commit
> is
> > made for this consumer group in the future, will this effectively
> > disable consumed log retention for this topic? In other words, how do
> this
> > KIP distinguish active consumer group from inactive ones?
> >
> > 2) Why do we need new configs such as log.retention.commitoffset.hours?
> Can
> >we simply delete log segments if consumed log retention is enabled for
> this
> > topic and all consumer groups have consumed messages in the log segment?
> >
> > Thanks,
> > Dong
> >
> >
> >
> >On Sat, Oct 8, 2016 at 2:15 AM, Pengwei (L) 
> wrote:
> >
> > > Hi Becket,
> > >
> > >   Thanks for the feedback:
> > > 1.  We use the simple consumer api to query the commit offset, so we
> don't
> > > need to specify the consumer group.
> > > 2.  Every broker using the simple consumer api(OffsetFetchKey) to query
> > > the commit offset in the log retention process.  The client can commit
> > > offset or not.
> > > 3.  It does not need to distinguish the follower brokers or leader
> > > brokers,  every brokers can query.
> > > 4.  We don't need to change the protocols, we mainly change the log
> > > retention process in the log manager.
> > >
> > >   One question is the query min offset need O(partitions * groups) time
> > > complexity, another alternative is to build an internal topic to save
> every
> > > partition's min offset, it can reduce to O(1).
> > > I will update the wiki for more details.
> > >
> > > Thanks,
> > > David
> > >
> > >
> > > > Hi Pengwei,
> > > >
> > > > Thanks for the KIP proposal. It is a very useful KIP. At a high
> level,
> > > the
> > > > proposed behavior looks reasonable to me.
> > > >
> > > > However, it seems that some of the details are not mentioned in the
> KIP.
> > > > For example,
> > > >
> > > > 1. How will the expected consumer group be specified? Is it through
> a per
> > > > topic dynamic configuration?
> > > > 2. How do the brokers detect the consumer offsets? Is it required
> for a
> > > > consumer to commit offsets?
> > > > 3. How do all the replicas know the about the committed offsets?
> e.g. 1)
> > > > non-coordinator brokers which do not have the committed offsets, 2)
> > > > follower brokers which do not have consumers directly consuming from
> it.
> > > > 4. Is there any other changes need to be made (e.g. new protocols) in
> > > > addition to the configuration change?
> > > >
> > > > It would be great if you can update the wiki to have more details.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Wed, Sep 7, 2016 at 2:26 AM, Pengwei (L) 
> > > wrote:
> > > >
> > > > > Hi All,
> > > > >I have made a KIP to enhance the log retention, 

[jira] [Created] (KAFKA-4282) View / Delete Replication Quotas via Config Command EntityName Wildcards for Topic/Broker

[Ben Stopford (JIRA)]

Ben Stopford created KAFKA-4282:
---

 Summary: View / Delete Replication Quotas via Config Command 
EntityName Wildcards for Topic/Broker
 Key: KAFKA-4282
 URL: https://issues.apache.org/jira/browse/KAFKA-4282
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.10.1.1
Reporter: Ben Stopford


It is likely that people will need a utility for checking if a quota is set and 
removing it. This would be useful if something untoward happened, say the 
throttle was not removed from a certain broker due to a network glitch at the 
time of removal. 

Currently it is possible to view and delete replication quota configs using the 
config command, but you'd need to script up something that called the command 
for each topic and each broker. It's also possible to delete the throttle for a 
assignment using the ReassignPartitionsCommand but that requires an outstanding 
assignment and matching json doc. 

Thus it would be good to add something to the ConfigCommand that wrapped this 
functionality. 

Probably the best way would be to simply add wildcard support to the EntityName 
field for Broker and Topic entity types. 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4273) Streams DSL - Add TTL / retention period support for intermediate topics and state stores

[Davor Poldrugo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Poldrugo updated KAFKA-4273:
--
Description: 
Hi!
I'm using Streams DSL (0.10.0.1), which can only use RocksDB for local state as 
far as I know - it's not configurable.
In my use case my data has TTL / retnetion period. It's 48 hours. After that - 
data can be discarded.

I join two topics: "messages" and "prices" using windowed inner join.
The two intermediate Kafka topics for this join are named:
 * messages-prices-join-this-changelog
 * messages-prices-join-other-changelog

Since these topics are created as compacted by Kafka Streams, and I don't wan't 
to keep data forever, I have altered them to not use compaction. Right now my 
RocksDB state stores grow indefinitely, and I don't have any options to define 
TTL, or somehow periodically clean the older data.

A "hack" that I use to keep my disk usage low - I have schedulled a job to 
periodically stop Kafka Streams instances - one at the time. This triggers a 
rebalance, and partitions migrate to other instances. When the instance is 
started again, there's another rebalance, and sometimes this instance starts 
processing partitions that wasn't processing before the stop - which leads to 
deletion of the RocksDB state store for those partitions 
(state.cleanup.delay.ms). In the next rebalance the local store is recreated 
with a restore consumer - which reads data from - as previously mentioned - a 
non compacted topic. And this effectively leads to a "hacked TTL support" in 
Kafka Streams DSL.

Questions:
 * Do you think would be reasonable to add support in the DSL api to define TTL 
for local store?
 * Which opens another question - there are use cases which don't need the 
intermediate topics to be created as "compact". Could also this be added to the 
DSL api? Maybe only this could be added, and this flag should also be used for 
the RocksDB TTL. Of course in this case another config would be mandatory - the 
retention period or TTL for the intermediate topics and the state stores. I saw 
there is a new cleanup.policy - compact_and_delete - added with KAFKA-4015.
 * Which also leads to another question, maybe some intermediate topics / state 
stores need different TTL, so a it's not as simple as that. But after 
KAFKA-3870, it will be easier.

RocksDB supports TTL:
 * 
https://github.com/apache/kafka/blob/0.10.0.1/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L166
 * https://github.com/facebook/rocksdb/wiki/Time-to-Live
 * 
https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/TtlDB.java

A somehow similar issue: KAFKA-4212



  was:
Hi!
I'm using Streams DSL (0.10.0.1), which can only use RocksDB for local state as 
far as I know - it's not configurable.
In my use case my data has TTL / retnetion period. It's 48 hours. After that - 
data can be discarded.

I join two topics: "messages" and "prices" using windowed inner join.
The two intermediate Kafka topics for this join are named:
 * messages-prices-join-this-changelog
 * messages-prices-join-other-changelog

Since these topics are created as compacted by Kafka Streams, and I don't wan't 
to keep data forever, I have altered them to not use compaction. Right now my 
RocksDB state stores grow indefinitely, and I don't have any options to define 
TTL, or somehow periodically clean the older data. I 

A "hack" that I use to keep my disk usage low - I have schedulled a job to 
periodically stop Kafka Streams instances - one at the time. This triggers a 
rebalance, and partitions migrate to other instances. When the instance is 
started again, there's another rebalance, and sometimes this instance starts 
processing partitions that wasn't processing before the stop - which leads to 
deletion of the RocksDB state store for those partitions 
(state.cleanup.delay.ms). In the next rebalance the local store is recreated 
with a restore consumer - which reads data from - as previously mentioned - a 
non compacted topic. And this effectively leads to a "hacked TTL support" in 
Kafka Streams DSL.

Questions:
 * Do you think would be reasonable to add support in the DSL api to define TTL 
for local store?
 * Which opens another question - there are use cases which don't need the 
intermediate topics to be created as "compact". Could also this be added to the 
DSL api? Maybe only this could be added, and this flag should also be used for 
the RocksDB TTL. Of course in this case another config would be mandatory - the 
retention period or TTL for the intermediate topics and the state stores. I saw 
there is a new cleanup.policy - compact_and_delete - added with KAFKA-4015.
 * Which also leads to another question, maybe some intermediate topics / state 
stores need different TTL, so a it's not as simple as that. But after 
KAFKA-3870, it will be easier.

RocksDB supports TTL:
 * 

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

[Pengwei (L)]

Hi Dong
   Thanks for the questions:

1.  Now we don't distinguish inactive or active groups. Because in some case 
maybe inactive group will become active again, and using the previous commit 
offset.

So we will not delete the log segment in the consumer retention if there are 
some groups consume but not commit, but the log segment can be delete by
 the force retention.

2.  These configs are used to determine the out of date time of the consumed 
retention, like the parameters of the force retention (log.retention.hours, 
log.retention.minutes, log.retention.ms). For example, users want the save the 
log for 3 days, after 3 days, kafka will delete the log segments which are

consumed by all the consumer group.  The log retention thread need these 
parameters.


Thanks,
David


> Hey David,
>
> Thanks for the KIP. Can you help with the following two questions:
>
> 1) If someone start a consumer (e.g. kafka-console-consumer) to consume a
> topic for debug/validation purpose, a randome consumer group may be created
> and offset may be committed for this consumer group. If no offset commit is
> made for this consumer group in the future, will this effectively
> disable consumed log retention for this topic? In other words, how do this
> KIP distinguish active consumer group from inactive ones?
>
> 2) Why do we need new configs such as log.retention.commitoffset.hours? Can
>we simply delete log segments if consumed log retention is enabled for this
> topic and all consumer groups have consumed messages in the log segment?
>
> Thanks,
> Dong
>
>
>
>On Sat, Oct 8, 2016 at 2:15 AM, Pengwei (L)  wrote:
>
> > Hi Becket,
> >
> >   Thanks for the feedback:
> > 1.  We use the simple consumer api to query the commit offset, so we don't
> > need to specify the consumer group.
> > 2.  Every broker using the simple consumer api(OffsetFetchKey) to query
> > the commit offset in the log retention process.  The client can commit
> > offset or not.
> > 3.  It does not need to distinguish the follower brokers or leader
> > brokers,  every brokers can query.
> > 4.  We don't need to change the protocols, we mainly change the log
> > retention process in the log manager.
> >
> >   One question is the query min offset need O(partitions * groups) time
> > complexity, another alternative is to build an internal topic to save every
> > partition's min offset, it can reduce to O(1).
> > I will update the wiki for more details.
> >
> > Thanks,
> > David
> >
> >
> > > Hi Pengwei,
> > >
> > > Thanks for the KIP proposal. It is a very useful KIP. At a high level,
> > the
> > > proposed behavior looks reasonable to me.
> > >
> > > However, it seems that some of the details are not mentioned in the KIP.
> > > For example,
> > >
> > > 1. How will the expected consumer group be specified? Is it through a per
> > > topic dynamic configuration?
> > > 2. How do the brokers detect the consumer offsets? Is it required for a
> > > consumer to commit offsets?
> > > 3. How do all the replicas know the about the committed offsets? e.g. 1)
> > > non-coordinator brokers which do not have the committed offsets, 2)
> > > follower brokers which do not have consumers directly consuming from it.
> > > 4. Is there any other changes need to be made (e.g. new protocols) in
> > > addition to the configuration change?
> > >
> > > It would be great if you can update the wiki to have more details.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Wed, Sep 7, 2016 at 2:26 AM, Pengwei (L) 
> > wrote:
> > >
> > > > Hi All,
> > > >I have made a KIP to enhance the log retention, details as follows:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 68+Add+a+consumed+log+retention+before+log+retention
> > > >Now start a discuss thread for this KIP , looking forward to the
> > > > feedback.
> > > >
> > > > Thanks,
> > > > David
> > > >
> > > >