[jira] [Commented] (KAFKA-2912) Add error code 4 (InvalidFetchSize) to Errors.java

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035872#comment-15035872
 ] 

jin xing commented on KAFKA-2912:
-

I believe the validation for fetch size should be done inside of KafkaApis : 
handleFetchRequets(request: RequestChannel.Request) ;
The question is how to define a fetch request is invalid?
Maybe we can judge by parameters: buffer size, fetchRequest.maxWait, 
fetchRequest.minBytes, quotas of throughput, request.timeout.ms...;
Is this point of view right?

> Add error code 4 (InvalidFetchSize) to Errors.java
> --
>
> Key: KAFKA-2912
> URL: https://issues.apache.org/jira/browse/KAFKA-2912
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>
> org.apache.kafka.common.protocol.Errors.java has:
> {quote}
> // TODO: errorCode 4 for InvalidFetchSize
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-2929) Remove duplicate error mapping functionality

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-2929 started by Grant Henke.
--
> Remove duplicate error mapping functionality
> 
>
> Key: KAFKA-2929
> URL: https://issues.apache.org/jira/browse/KAFKA-2929
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> Kafka common and core both have a class that maps error codes and exceptions. 
> To prevent errors and issues with consistency we should remove 
> ErrorMapping.scala in core in favor or Errors.java in common. Any duplicated 
> exceptions in core should be removed as well to ensure the mapping is correct.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #198

[Apache Jenkins Server]

See 

Changes:

[junrao] KAFKA-2642; Run replication tests with SSL and SASL clients

--
[...truncated 2833 lines...]

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.log.OffsetIndexTest > lookupExtremeCases PASSED

kafka.log.OffsetIndexTest > appendTooMany PASSED

kafka.log.OffsetIndexTest > randomLookupTest PASSED

kafka.log.OffsetIndexTest > testReopen PASSED

kafka.log.OffsetIndexTest > appendOutOfOrder PASSED

kafka.log.OffsetIndexTest > truncate PASSED

kafka.log.LogSegmentTest > testRecoveryWithCorruptMessage PASSED

kafka.log.LogSegmentTest > testRecoveryFixesCorruptIndex PASSED

kafka.log.LogSegmentTest > testReadFromGap PASSED

kafka.log.LogSegmentTest > testTruncate PASSED

kafka.log.LogSegmentTest > testReadBeforeFirstOffset PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeAppendMessage PASSED

kafka.log.LogSegmentTest > testChangeFileSuffixes PASSED

kafka.log.LogSegmentTest > testMaxOffset PASSED

kafka.log.LogSegmentTest > testNextOffsetCalculation PASSED

kafka.log.LogSegmentTest > testReadOnEmptySegment PASSED

kafka.log.LogSegmentTest > testReadAfterLast PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeClearShutdown PASSED

kafka.log.LogSegmentTest > testTruncateFull PASSED

kafka.log.CleanerTest > testBuildOffsetMap PASSED

kafka.log.CleanerTest > testSegmentGrouping PASSED

kafka.log.CleanerTest > testCleanSegmentsWithAbort PASSED

kafka.log.CleanerTest > testSegmentGroupingWithSparseOffsets PASSED

kafka.log.CleanerTest > testRecoveryAfterCrash PASSED

kafka.log.CleanerTest > testLogToClean PASSED

kafka.log.CleanerTest > testCleaningWithDeletes PASSED

kafka.log.CleanerTest > testCleanSegments PASSED

kafka.log.CleanerTest > testCleaningWithUnkeyedMessages PASSED

kafka.log.OffsetMapTest > testClear PASSED

kafka.log.OffsetMapTest > testBasicValidation PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupStable PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatIllegalGeneration 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testDescribeGroupWrongCoordinator PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupRebalancing 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaderFailureInSyncGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testGenerationIdIncrementsOnRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFromIllegalGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testInvalidGroupId PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesStableGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatDuringRebalanceCausesRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentGroupProtocol PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooLarge PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooSmall PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupEmptyAssignment 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetWithDefaultGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupFromUnchangedLeaderShouldRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesRebalancingGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFollowerAfterLeader PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testCommitOffsetInAwaitingSync 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testJoinGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupFromUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentProtocolType PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetFromUnknownGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testLeaveGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerNewGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupFromUnchangedFollowerDoesNotRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest 

[jira] [Created] (KAFKA-2929) Remove duplicate error mapping functionality

[Grant Henke (JIRA)]

Grant Henke created KAFKA-2929:
--

 Summary: Remove duplicate error mapping functionality
 Key: KAFKA-2929
 URL: https://issues.apache.org/jira/browse/KAFKA-2929
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.9.0.0
Reporter: Grant Henke
Assignee: Grant Henke


Kafka common and core both have a class that maps error codes and exceptions. 
To prevent errors and issues with consistency we should remove 
ErrorMapping.scala in core in favor or Errors.java in common. Any duplicated 
exceptions in core should be removed as well to ensure the mapping is correct.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1894) Avoid long or infinite blocking in the consumer

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036256#comment-15036256
 ] 

Guozhang Wang commented on KAFKA-1894:
--

[~BigAndy] You can indeed interrupt the poll() call by calling wakeup() from 
another thread (search for the paragraph of "multi-threaded processing"):

http://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

> Avoid long or infinite blocking in the consumer
> ---
>
> Key: KAFKA-1894
> URL: https://issues.apache.org/jira/browse/KAFKA-1894
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>Assignee: Jason Gustafson
> Fix For: 0.10.0.0
>
>
> The new consumer has a lot of loops that look something like
> {code}
>   while(!isThingComplete())
> client.poll();
> {code}
> This occurs both in KafkaConsumer but also in NetworkClient.completeAll. 
> These retry loops are actually mostly the behavior we want but there are 
> several cases where they may cause problems:
>  - In the case of a hard failure we may hang for a long time or indefinitely 
> before realizing the connection is lost.
>  - In the case where the cluster is malfunctioning or down we may retry 
> forever.
> It would probably be better to give a timeout to these. The proposed approach 
> would be to add something like retry.time.ms=6 and only continue retrying 
> for that period of time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2930) Update references to ZooKeeper in the docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036238#comment-15036238
 ] 

ASF GitHub Bot commented on KAFKA-2930:
---

GitHub user fpj opened a pull request:

https://github.com/apache/kafka/pull/615

KAFKA-2930: Update references to ZooKeeper in the docs.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fpj/kafka KAFKA-2930

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/615.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #615


commit 312e885390bd665cd349408df8f28fad20cca872
Author: Flavio Junqueira 
Date:   2015-12-02T17:44:13Z

KAFKA-2930: Updated doc.




> Update references to ZooKeeper in the docs
> --
>
> Key: KAFKA-2930
> URL: https://issues.apache.org/jira/browse/KAFKA-2930
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Flavio Junqueira
>Assignee: Flavio Junqueira
> Fix For: 0.9.0.1
>
>
> Information about ZooKeeper in the ops doc is stale, it refers to branch 3.3 
> and Kafka is already using branch 3.4.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-2851) system tests: error copying keytab file

[Anna Povzner (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010231#comment-15010231
 ] 

Anna Povzner edited comment on KAFKA-2851 at 12/2/15 6:16 PM:
--

Pull request: https://github.com/apache/kafka/pull/610




was (Author: apovzner):
Pull request: https://github.com/apache/kafka/pull/609



> system tests: error copying keytab file
> ---
>
> Key: KAFKA-2851
> URL: https://issues.apache.org/jira/browse/KAFKA-2851
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Anna Povzner
>Priority: Minor
>
> It is best to use unique paths for temporary files on the test driver machine 
> so that multiple test jobs don't conflict. 
> If the test driver machine is running multiple ducktape jobs concurrently, as 
> is the case with Confluent nightly test runs, conflicts can occur if the same 
> canonical path is always used.
> In this case, security_config.py copies a file to /tmp/keytab on the test 
> driver machine, while other jobs may remove this from the driver machine. 
> Then you can get errors like this:
> {code}
> 
> test_id:
> 2015-11-17--001.kafkatest.tests.replication_test.ReplicationTest.test_replication_with_broker_failure.security_protocol=SASL_PLAINTEXT.failure_mode=clean_bounce
> status: FAIL
> run time:   1 minute 33.395 seconds
> 
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/tests/runner.py",
>  line 101, in run_all_tests
> result.data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/tests/runner.py",
>  line 151, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 132, in test_replication_with_broker_failure
> self.run_produce_consume_validate(core_test_action=lambda: 
> failures[failure_mode](self))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 66, in run_produce_consume_validate
> core_test_action()
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 132, in 
> self.run_produce_consume_validate(core_test_action=lambda: 
> failures[failure_mode](self))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 43, in clean_bounce
> test.kafka.restart_node(prev_leader_node, clean_shutdown=True)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/kafka/kafka.py",
>  line 275, in restart_node
> self.start_node(node)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/kafka/kafka.py",
>  line 123, in start_node
> self.security_config.setup_node(node)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/security/security_config.py",
>  line 130, in setup_node
> node.account.scp_to(MiniKdc.LOCAL_KEYTAB_FILE, SecurityConfig.KEYTAB_PATH)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/cluster/remoteaccount.py",
>  line 174, in scp_to
> return self._ssh_quiet(self.scp_to_command(src, dest, recursive))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/cluster/remoteaccount.py",
>  line 219, in _ssh_quiet
> raise e
> CalledProcessError: Command 'scp -o 'HostName 52.33.250.202' -o 'Port 22' -o 
> 'UserKnownHostsFile /dev/null' -o 'StrictHostKeyChecking no' -o 
> 'PasswordAuthentication no' -o 'IdentityFile /var/lib/jenkins/muckrake.pem' 
> -o 'IdentitiesOnly yes' -o 'LogLevel FATAL'  /tmp/keytab 
> ubuntu@worker2:/mnt/security/keytab' returned non-zero exit status 1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2851) system tests: error copying keytab file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036304#comment-15036304
 ] 

ASF GitHub Bot commented on KAFKA-2851:
---

Github user apovzner closed the pull request at:

https://github.com/apache/kafka/pull/609


> system tests: error copying keytab file
> ---
>
> Key: KAFKA-2851
> URL: https://issues.apache.org/jira/browse/KAFKA-2851
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Anna Povzner
>Priority: Minor
>
> It is best to use unique paths for temporary files on the test driver machine 
> so that multiple test jobs don't conflict. 
> If the test driver machine is running multiple ducktape jobs concurrently, as 
> is the case with Confluent nightly test runs, conflicts can occur if the same 
> canonical path is always used.
> In this case, security_config.py copies a file to /tmp/keytab on the test 
> driver machine, while other jobs may remove this from the driver machine. 
> Then you can get errors like this:
> {code}
> 
> test_id:
> 2015-11-17--001.kafkatest.tests.replication_test.ReplicationTest.test_replication_with_broker_failure.security_protocol=SASL_PLAINTEXT.failure_mode=clean_bounce
> status: FAIL
> run time:   1 minute 33.395 seconds
> 
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/tests/runner.py",
>  line 101, in run_all_tests
> result.data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/tests/runner.py",
>  line 151, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 132, in test_replication_with_broker_failure
> self.run_produce_consume_validate(core_test_action=lambda: 
> failures[failure_mode](self))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 66, in run_produce_consume_validate
> core_test_action()
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 132, in 
> self.run_produce_consume_validate(core_test_action=lambda: 
> failures[failure_mode](self))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 43, in clean_bounce
> test.kafka.restart_node(prev_leader_node, clean_shutdown=True)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/kafka/kafka.py",
>  line 275, in restart_node
> self.start_node(node)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/kafka/kafka.py",
>  line 123, in start_node
> self.security_config.setup_node(node)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/security/security_config.py",
>  line 130, in setup_node
> node.account.scp_to(MiniKdc.LOCAL_KEYTAB_FILE, SecurityConfig.KEYTAB_PATH)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/cluster/remoteaccount.py",
>  line 174, in scp_to
> return self._ssh_quiet(self.scp_to_command(src, dest, recursive))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/cluster/remoteaccount.py",
>  line 219, in _ssh_quiet
> raise e
> CalledProcessError: Command 'scp -o 'HostName 52.33.250.202' -o 'Port 22' -o 
> 'UserKnownHostsFile /dev/null' -o 'StrictHostKeyChecking no' -o 
> 'PasswordAuthentication no' -o 'IdentityFile /var/lib/jenkins/muckrake.pem' 
> -o 'IdentitiesOnly yes' -o 'LogLevel FATAL'  /tmp/keytab 
> ubuntu@worker2:/mnt/security/keytab' returned non-zero exit status 1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2930: Update references to ZooKeeper in ...

[fpj]

GitHub user fpj opened a pull request:

https://github.com/apache/kafka/pull/615

KAFKA-2930: Update references to ZooKeeper in the docs.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fpj/kafka KAFKA-2930

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/615.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #615


commit 312e885390bd665cd349408df8f28fad20cca872
Author: Flavio Junqueira 
Date:   2015-12-02T17:44:13Z

KAFKA-2930: Updated doc.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: Kafka-2310: Add config to prevent broker becom...

[abiletskyi]

GitHub user abiletskyi opened a pull request:

https://github.com/apache/kafka/pull/614

Kafka-2310: Add config to prevent broker becoming controller



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/stealthly/kafka KAFKA-2310-trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/614.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #614


commit ec16ef8147e9de7878e95dad2d461284e4271fbe
Author: Joe Stein 
Date:   2015-08-17T10:41:46Z

Merge pull request #1 from apache/trunk

upstream fixes

commit 3f88d933f4cfbd57727fd0a6afe4e368ca8392fc
Author: Joe Stein 
Date:   2015-09-18T12:11:19Z

Merge pull request #2 from apache/trunk

upstream updates

commit 5f9aeb5cb492c5ef07e799d2120cad818a7fa8f1
Author: abiletskyi 
Date:   2015-12-02T15:04:17Z

Merge pull request #3 from apache/trunk

upstream updates

commit 719942cda5d6d12d2afe2271aac91582eab6ce7a
Author: Andrii Biletskyi 
Date:   2015-12-02T17:31:30Z

KAFKA-2310 - Add config to prevent broker becoming controller




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-2930) Update references to ZooKeeper in the docs

[Flavio Junqueira (JIRA)]

Flavio Junqueira created KAFKA-2930:
---

 Summary: Update references to ZooKeeper in the docs
 Key: KAFKA-2930
 URL: https://issues.apache.org/jira/browse/KAFKA-2930
 Project: Kafka
  Issue Type: Improvement
Reporter: Flavio Junqueira
Assignee: Flavio Junqueira
 Fix For: 0.9.0.1


Information about ZooKeeper in the ops doc is stale, it refers to branch 3.3 
and Kafka is already using branch 3.4.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2851 Using random dir under /temp for lo...

[apovzner]

Github user apovzner closed the pull request at:

https://github.com/apache/kafka/pull/609


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (KAFKA-1894) Avoid long or infinite blocking in the consumer

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036256#comment-15036256
 ] 

Guozhang Wang edited comment on KAFKA-1894 at 12/2/15 5:56 PM:
---

[~BigAndy] You can indeed interrupt the poll() call by calling wakeup() from 
another thread (search for the paragraph of "multi-threaded processing"):

http://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

This ticket is for resolving the issue that poll(timeout) can actually block 
longer than the specified timeout value if the broker is not available and no 
one else wakes it up.


was (Author: guozhang):
[~BigAndy] You can indeed interrupt the poll() call by calling wakeup() from 
another thread (search for the paragraph of "multi-threaded processing"):

http://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

> Avoid long or infinite blocking in the consumer
> ---
>
> Key: KAFKA-1894
> URL: https://issues.apache.org/jira/browse/KAFKA-1894
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>Assignee: Jason Gustafson
> Fix For: 0.10.0.0
>
>
> The new consumer has a lot of loops that look something like
> {code}
>   while(!isThingComplete())
> client.poll();
> {code}
> This occurs both in KafkaConsumer but also in NetworkClient.completeAll. 
> These retry loops are actually mostly the behavior we want but there are 
> several cases where they may cause problems:
>  - In the case of a hard failure we may hang for a long time or indefinitely 
> before realizing the connection is lost.
>  - In the case where the cluster is malfunctioning or down we may retry 
> forever.
> It would probably be better to give a timeout to these. The proposed approach 
> would be to add something like retry.time.ms=6 and only continue retrying 
> for that period of time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2851 Using random file names for local k...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/610


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-2932) Adjust importance level of Kafka Connect configs

[Ewen Cheslack-Postava (JIRA)]

Ewen Cheslack-Postava created KAFKA-2932:


 Summary: Adjust importance level of Kafka Connect configs
 Key: KAFKA-2932
 URL: https://issues.apache.org/jira/browse/KAFKA-2932
 Project: Kafka
  Issue Type: Bug
  Components: copycat
Affects Versions: 0.9.0.0
Reporter: Ewen Cheslack-Postava
Assignee: Ewen Cheslack-Postava


Some of the configuration importance levels are out of whack, probably due to 
the way they evolved over time. For example, the internal converter settings 
are currently marked with high importance, but they are really an internal 
implementation detail that the user usually shouldn't need to worry about.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2929: Remove duplicate error mapping fun...

[granthenke]

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/616

KAFKA-2929: Remove duplicate error mapping functionality

Removes ErrorMapping.scala in core in favor or Errors.java in common. 
Duplicated exceptions in core are removed as well, to ensure the mapping is 
correct.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka error-mapping

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/616.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #616


commit 10003b33140144f5bd97ba37654e8396db724d92
Author: Grant Henke 
Date:   2015-12-02T16:04:59Z

KAFKA-2929: Remove duplicate error mapping functionality




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-2880: consumer should handle disconnect/...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/581


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-2880) Fetcher.getTopicMetadata NullPointerException when broker cannot be reached

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-2880.
--
   Resolution: Fixed
Fix Version/s: 0.9.0.1

Issue resolved by pull request 581
[https://github.com/apache/kafka/pull/581]

> Fetcher.getTopicMetadata NullPointerException when broker cannot be reached
> ---
>
> Key: KAFKA-2880
> URL: https://issues.apache.org/jira/browse/KAFKA-2880
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
> Fix For: 0.9.0.1
>
>
> The Fetcher class will throw a NullPointerException if a broker cannot be 
> reached:
> {quote}
> Exception in thread "main" java.lang.NullPointerException
> at 
> org.apache.kafka.common.requests.MetadataResponse.(MetadataResponse.java:130)
> at 
> org.apache.kafka.clients.consumer.internals.Fetcher.getTopicMetadata(Fetcher.java:203)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:1143)
> at 
> org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:126)
> at 
> org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:85)
> at org.apache.kafka.connect.runtime.Worker.start(Worker.java:108)
> at org.apache.kafka.connect.runtime.Connect.start(Connect.java:56)
> at 
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:62)
> {quote}
> This is trivially reproduced by trying to start Kafka Connect in distributed 
> mode (i.e. connect-distributed.sh config/connect-distributed.properties) with 
> no broker running. However, it's not specific to Kafka Connect, it just 
> happens to use the consumer in a way that triggers it reliably.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: Minor: ConsoleConsumer - Fix number of process...

[luafran]

GitHub user luafran opened a pull request:

https://github.com/apache/kafka/pull/617

Minor: ConsoleConsumer - Fix number of processed messages count

kafka-console-consumer.sh is showing an incorrect number of
messages processed, counting one more message than the actual
number of processed messages.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/luafran/kafka 
console-consumer-number-of-processed-messages

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/617.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #617


commit f5e8fd48ac81de61a14ff3f1d54e0bf3bd392718
Author: Luciano Afranllie 
Date:   2015-12-02T19:27:28Z

Minor: ConsoleConsumer - Fix number of processed messages count

kafka-console-consumer.sh is showing an incorrect number of
messages processed, counting one more message than the actual
number of processed messages.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka_0.9.0_jdk7 #54

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-2880: consumer should handle disconnect/timeout for metadata

--
[...truncated 3800 lines...]
kafka.log.LogTest > testParseTopicPartitionNameForMissingTopic PASSED

kafka.log.LogTest > testIndexRebuild PASSED

kafka.log.LogTest > testLogRolls PASSED

kafka.log.LogTest > testMessageSizeCheck PASSED

kafka.log.LogTest > testAsyncDelete PASSED

kafka.log.LogTest > testReadOutOfRange PASSED

kafka.log.LogTest > testReadAtLogGap PASSED

kafka.log.LogTest > testTimeBasedLogRoll PASSED

kafka.log.LogTest > testLoadEmptyLog PASSED

kafka.log.LogTest > testMessageSetSizeCheck PASSED

kafka.log.LogTest > testIndexResizingAtTruncation PASSED

kafka.log.LogTest > testCompactedTopicConstraints PASSED

kafka.log.LogTest > testThatGarbageCollectingSegmentsDoesntChangeOffset PASSED

kafka.log.LogTest > testAppendAndReadWithSequentialOffsets PASSED

kafka.log.LogTest > testParseTopicPartitionNameForNull PASSED

kafka.log.LogTest > testAppendAndReadWithNonSequentialOffsets PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingSeparator PASSED

kafka.log.LogTest > testCorruptIndexRebuild PASSED

kafka.log.LogTest > testBogusIndexSegmentsAreRemoved PASSED

kafka.log.LogTest > testCompressedMessages PASSED

kafka.log.LogTest > testAppendMessageWithNullPayload PASSED

kafka.log.LogTest > testCorruptLog PASSED

kafka.log.LogTest > testLogRecoversToCorrectOffset PASSED

kafka.log.LogTest > testReopenThenTruncate PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition PASSED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName PASSED

kafka.log.LogTest > testOpenDeletesObsoleteFiles PASSED

kafka.log.LogTest > testSizeBasedLogRoll PASSED

kafka.log.LogTest > testTimeBasedLogRollJitter PASSED

kafka.log.LogTest > testParseTopicPartitionName PASSED

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.log.LogConfigTest > testFromPropsEmpty PASSED

kafka.log.LogConfigTest > testKafkaConfigToProps PASSED

kafka.log.LogConfigTest > testFromPropsInvalid PASSED

kafka.log.CleanerTest > testBuildOffsetMap PASSED

kafka.log.CleanerTest > testSegmentGrouping PASSED

kafka.log.CleanerTest > testCleanSegmentsWithAbort PASSED

kafka.log.CleanerTest > testSegmentGroupingWithSparseOffsets PASSED

kafka.log.CleanerTest > testRecoveryAfterCrash PASSED

kafka.log.CleanerTest > testLogToClean PASSED

kafka.log.CleanerTest > testCleaningWithDeletes PASSED

kafka.log.CleanerTest > testCleanSegments PASSED

kafka.log.CleanerTest > testCleaningWithUnkeyedMessages PASSED

kafka.log.OffsetIndexTest > lookupExtremeCases PASSED

kafka.log.OffsetIndexTest > appendTooMany PASSED

kafka.log.OffsetIndexTest > randomLookupTest PASSED

kafka.log.OffsetIndexTest > testReopen PASSED

kafka.log.OffsetIndexTest > appendOutOfOrder PASSED

kafka.log.OffsetIndexTest > truncate PASSED

639 tests completed, 1 failed
:kafka_0.9.0_jdk7:core:test FAILED
:test_core_2_10_5 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':core:test'.
> There were failing tests. See the report at: 
> file://

* Try:
Run with --info or --debug option to get more log output.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
':core:test'.
at 
org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:69)
at 
org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:46)
at 
org.gradle.api.internal.tasks.execution.PostExecutionAnalysisTaskExecuter.execute(PostExecutionAnalysisTaskExecuter.java:35)
at 
org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:64)
at 
org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
at 
org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:52)
at 
org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:52)
at 
org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:53)
at 
org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
at 
org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:203)
at 
org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:185)
at 

[jira] [Commented] (KAFKA-2880) Fetcher.getTopicMetadata NullPointerException when broker cannot be reached

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036424#comment-15036424
 ] 

ASF GitHub Bot commented on KAFKA-2880:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/581


> Fetcher.getTopicMetadata NullPointerException when broker cannot be reached
> ---
>
> Key: KAFKA-2880
> URL: https://issues.apache.org/jira/browse/KAFKA-2880
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
>
> The Fetcher class will throw a NullPointerException if a broker cannot be 
> reached:
> {quote}
> Exception in thread "main" java.lang.NullPointerException
> at 
> org.apache.kafka.common.requests.MetadataResponse.(MetadataResponse.java:130)
> at 
> org.apache.kafka.clients.consumer.internals.Fetcher.getTopicMetadata(Fetcher.java:203)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:1143)
> at 
> org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:126)
> at 
> org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:85)
> at org.apache.kafka.connect.runtime.Worker.start(Worker.java:108)
> at org.apache.kafka.connect.runtime.Connect.start(Connect.java:56)
> at 
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:62)
> {quote}
> This is trivially reproduced by trying to start Kafka Connect in distributed 
> mode (i.e. connect-distributed.sh config/connect-distributed.properties) with 
> no broker running. However, it's not specific to Kafka Connect, it just 
> happens to use the consumer in a way that triggers it reliably.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2929) Remove duplicate error mapping functionality

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036447#comment-15036447
 ] 

ASF GitHub Bot commented on KAFKA-2929:
---

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/616

KAFKA-2929: Remove duplicate error mapping functionality

Removes ErrorMapping.scala in core in favor or Errors.java in common. 
Duplicated exceptions in core are removed as well, to ensure the mapping is 
correct.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka error-mapping

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/616.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #616


commit 10003b33140144f5bd97ba37654e8396db724d92
Author: Grant Henke 
Date:   2015-12-02T16:04:59Z

KAFKA-2929: Remove duplicate error mapping functionality




> Remove duplicate error mapping functionality
> 
>
> Key: KAFKA-2929
> URL: https://issues.apache.org/jira/browse/KAFKA-2929
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> Kafka common and core both have a class that maps error codes and exceptions. 
> To prevent errors and issues with consistency we should remove 
> ErrorMapping.scala in core in favor or Errors.java in common. Any duplicated 
> exceptions in core should be removed as well to ensure the mapping is correct.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2933) Failure in kafka.api.PlaintextConsumerTest.testMultiConsumerDefaultAssignment

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-2933:


 Summary: Failure in 
kafka.api.PlaintextConsumerTest.testMultiConsumerDefaultAssignment
 Key: KAFKA-2933
 URL: https://issues.apache.org/jira/browse/KAFKA-2933
 Project: Kafka
  Issue Type: Sub-task
Affects Versions: 0.9.1.0
Reporter: Guozhang Wang


{code}
kafka.api.PlaintextConsumerTest > testMultiConsumerDefaultAssignment FAILED
java.lang.AssertionError: Did not get valid assignment for partitions 
[topic1-2, topic2-0, topic1-4, topic-1, topic-0, topic2-1, topic1-0, topic1-3, 
topic1-1, topic2-2] after we changed subscription
at org.junit.Assert.fail(Assert.java:88)
at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:747)
at 
kafka.api.PlaintextConsumerTest.validateGroupAssignment(PlaintextConsumerTest.scala:644)
at 
kafka.api.PlaintextConsumerTest.changeConsumerGroupSubscriptionAndValidateAssignment(PlaintextConsumerTest.scala:663)
at 
kafka.api.PlaintextConsumerTest.testMultiConsumerDefaultAssignment(PlaintextConsumerTest.scala:461)

java.util.ConcurrentModificationException: KafkaConsumer is not safe for 
multi-threaded access
{code}

Example: https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/1582/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2931) Consumer rolling upgrade test case

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-2931:
--

 Summary: Consumer rolling upgrade test case
 Key: KAFKA-2931
 URL: https://issues.apache.org/jira/browse/KAFKA-2931
 Project: Kafka
  Issue Type: Test
Reporter: Jason Gustafson
Assignee: Jason Gustafson


We need a system test which covers the rolling upgrade process for the new 
consumer. The idea is to start the consumers with a "range" assignment strategy 
and then upgrade to "round-robin" without any down-time. This validates the 
coordinator's protocol selection process.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2851) system tests: error copying keytab file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036413#comment-15036413
 ] 

ASF GitHub Bot commented on KAFKA-2851:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/610


> system tests: error copying keytab file
> ---
>
> Key: KAFKA-2851
> URL: https://issues.apache.org/jira/browse/KAFKA-2851
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Anna Povzner
>Priority: Minor
>
> It is best to use unique paths for temporary files on the test driver machine 
> so that multiple test jobs don't conflict. 
> If the test driver machine is running multiple ducktape jobs concurrently, as 
> is the case with Confluent nightly test runs, conflicts can occur if the same 
> canonical path is always used.
> In this case, security_config.py copies a file to /tmp/keytab on the test 
> driver machine, while other jobs may remove this from the driver machine. 
> Then you can get errors like this:
> {code}
> 
> test_id:
> 2015-11-17--001.kafkatest.tests.replication_test.ReplicationTest.test_replication_with_broker_failure.security_protocol=SASL_PLAINTEXT.failure_mode=clean_bounce
> status: FAIL
> run time:   1 minute 33.395 seconds
> 
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/tests/runner.py",
>  line 101, in run_all_tests
> result.data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/tests/runner.py",
>  line 151, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 132, in test_replication_with_broker_failure
> self.run_produce_consume_validate(core_test_action=lambda: 
> failures[failure_mode](self))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 66, in run_produce_consume_validate
> core_test_action()
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 132, in 
> self.run_produce_consume_validate(core_test_action=lambda: 
> failures[failure_mode](self))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 43, in clean_bounce
> test.kafka.restart_node(prev_leader_node, clean_shutdown=True)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/kafka/kafka.py",
>  line 275, in restart_node
> self.start_node(node)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/kafka/kafka.py",
>  line 123, in start_node
> self.security_config.setup_node(node)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/security/security_config.py",
>  line 130, in setup_node
> node.account.scp_to(MiniKdc.LOCAL_KEYTAB_FILE, SecurityConfig.KEYTAB_PATH)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/cluster/remoteaccount.py",
>  line 174, in scp_to
> return self._ssh_quiet(self.scp_to_command(src, dest, recursive))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/cluster/remoteaccount.py",
>  line 219, in _ssh_quiet
> raise e
> CalledProcessError: Command 'scp -o 'HostName 52.33.250.202' -o 'Port 22' -o 
> 'UserKnownHostsFile /dev/null' -o 'StrictHostKeyChecking no' -o 
> 'PasswordAuthentication no' -o 'IdentityFile /var/lib/jenkins/muckrake.pem' 
> -o 'IdentitiesOnly yes' -o 'LogLevel FATAL'  /tmp/keytab 
> ubuntu@worker2:/mnt/security/keytab' returned non-zero exit status 1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2933) Failure in kafka.api.PlaintextConsumerTest.testMultiConsumerDefaultAssignment

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson reassigned KAFKA-2933:
--

Assignee: Jason Gustafson

> Failure in kafka.api.PlaintextConsumerTest.testMultiConsumerDefaultAssignment
> -
>
> Key: KAFKA-2933
> URL: https://issues.apache.org/jira/browse/KAFKA-2933
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.1.0
>Reporter: Guozhang Wang
>Assignee: Jason Gustafson
>
> {code}
> kafka.api.PlaintextConsumerTest > testMultiConsumerDefaultAssignment FAILED
> java.lang.AssertionError: Did not get valid assignment for partitions 
> [topic1-2, topic2-0, topic1-4, topic-1, topic-0, topic2-1, topic1-0, 
> topic1-3, topic1-1, topic2-2] after we changed subscription
> at org.junit.Assert.fail(Assert.java:88)
> at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:747)
> at 
> kafka.api.PlaintextConsumerTest.validateGroupAssignment(PlaintextConsumerTest.scala:644)
> at 
> kafka.api.PlaintextConsumerTest.changeConsumerGroupSubscriptionAndValidateAssignment(PlaintextConsumerTest.scala:663)
> at 
> kafka.api.PlaintextConsumerTest.testMultiConsumerDefaultAssignment(PlaintextConsumerTest.scala:461)
> java.util.ConcurrentModificationException: KafkaConsumer is not safe for 
> multi-threaded access
> {code}
> Example: https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/1582/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2310) Add config to prevent broker becoming controller

[Aditya Auradkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036359#comment-15036359
 ] 

Aditya Auradkar commented on KAFKA-2310:


[~abiletskyi] - I see you submitted a pull request for this recently. 
https://github.com/apache/kafka/pull/614/files

Can you actually elaborate on the reasoning behind this change a bit more? I 
actually think we need a KIP to discuss this at least (given that it adds a new 
config). I'm not really sure preventing a broker from becoming a controller 
solves the underlying problem of a broker being overloaded.

> Add config to prevent broker becoming controller
> 
>
> Key: KAFKA-2310
> URL: https://issues.apache.org/jira/browse/KAFKA-2310
> Project: Kafka
>  Issue Type: Bug
>Reporter: Andrii Biletskyi
>Assignee: Andrii Biletskyi
> Attachments: KAFKA-2310.patch, KAFKA-2310_0.8.1.patch, 
> KAFKA-2310_0.8.2.patch
>
>
> The goal is to be able to specify which cluster brokers can serve as a 
> controller and which cannot. This way it will be possible to "reserve" 
> particular, not overloaded with partitions and other operations, broker as 
> controller.
> Proposed to add config _controller.eligibility_ defaulted to true (for 
> backward compatibility, since now any broker can become a controller)
> Patch will be available for trunk, 0.8.2 and 0.8.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2914) Kafka Connect Source connector for HBase

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036403#comment-15036403
 ] 

James Cheng commented on KAFKA-2914:


[~ewencp], do you want to fork/update 
https://github.com/wushujames/copycat-connector-skeleton? It hasn't yet been 
updated to support 0.9.0.

> Kafka Connect Source connector for HBase 
> -
>
> Key: KAFKA-2914
> URL: https://issues.apache.org/jira/browse/KAFKA-2914
> Project: Kafka
>  Issue Type: New Feature
>  Components: copycat
>Reporter: Niels Basjes
>Assignee: Ewen Cheslack-Postava
>
> In many cases I see HBase being used to persist data.
> I would like to listen to the changes and process them in a streaming system 
> (like Apache Flink).
> Feature request: A Kafka Connect "Source" that listens to the changes in a 
> specified HBase table. These changes are then stored in a 'standardized' form 
> in Kafka so that it becomes possible to process the observed changes in 
> near-realtime. I expect this 'standard' to be very HBase specific.
> Implementation suggestion: Perhaps listening to the HBase WAL like the "HBase 
> Side Effects Processor" does?
> https://github.com/NGDATA/hbase-indexer/tree/master/hbase-sep



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2929) Remove duplicate error mapping functionality

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-2929:
---
Status: Patch Available  (was: In Progress)

> Remove duplicate error mapping functionality
> 
>
> Key: KAFKA-2929
> URL: https://issues.apache.org/jira/browse/KAFKA-2929
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> Kafka common and core both have a class that maps error codes and exceptions. 
> To prevent errors and issues with consistency we should remove 
> ErrorMapping.scala in core in favor or Errors.java in common. Any duplicated 
> exceptions in core should be removed as well to ensure the mapping is correct.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2851) system tests: error copying keytab file

[Anna Povzner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Povzner updated KAFKA-2851:

Reviewer: Guozhang Wang  (was: Gwen Shapira)

> system tests: error copying keytab file
> ---
>
> Key: KAFKA-2851
> URL: https://issues.apache.org/jira/browse/KAFKA-2851
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Anna Povzner
>Priority: Minor
>
> It is best to use unique paths for temporary files on the test driver machine 
> so that multiple test jobs don't conflict. 
> If the test driver machine is running multiple ducktape jobs concurrently, as 
> is the case with Confluent nightly test runs, conflicts can occur if the same 
> canonical path is always used.
> In this case, security_config.py copies a file to /tmp/keytab on the test 
> driver machine, while other jobs may remove this from the driver machine. 
> Then you can get errors like this:
> {code}
> 
> test_id:
> 2015-11-17--001.kafkatest.tests.replication_test.ReplicationTest.test_replication_with_broker_failure.security_protocol=SASL_PLAINTEXT.failure_mode=clean_bounce
> status: FAIL
> run time:   1 minute 33.395 seconds
> 
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/tests/runner.py",
>  line 101, in run_all_tests
> result.data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/tests/runner.py",
>  line 151, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 132, in test_replication_with_broker_failure
> self.run_produce_consume_validate(core_test_action=lambda: 
> failures[failure_mode](self))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 66, in run_produce_consume_validate
> core_test_action()
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 132, in 
> self.run_produce_consume_validate(core_test_action=lambda: 
> failures[failure_mode](self))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 43, in clean_bounce
> test.kafka.restart_node(prev_leader_node, clean_shutdown=True)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/kafka/kafka.py",
>  line 275, in restart_node
> self.start_node(node)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/kafka/kafka.py",
>  line 123, in start_node
> self.security_config.setup_node(node)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/security/security_config.py",
>  line 130, in setup_node
> node.account.scp_to(MiniKdc.LOCAL_KEYTAB_FILE, SecurityConfig.KEYTAB_PATH)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/cluster/remoteaccount.py",
>  line 174, in scp_to
> return self._ssh_quiet(self.scp_to_command(src, dest, recursive))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/cluster/remoteaccount.py",
>  line 219, in _ssh_quiet
> raise e
> CalledProcessError: Command 'scp -o 'HostName 52.33.250.202' -o 'Port 22' -o 
> 'UserKnownHostsFile /dev/null' -o 'StrictHostKeyChecking no' -o 
> 'PasswordAuthentication no' -o 'IdentityFile /var/lib/jenkins/muckrake.pem' 
> -o 'IdentitiesOnly yes' -o 'LogLevel FATAL'  /tmp/keytab 
> ubuntu@worker2:/mnt/security/keytab' returned non-zero exit status 1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2718: Add logging to investigate intermi...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/613

KAFKA-2718: Add logging to investigate intermittent unit test failures

Print port and directories used by zookeeper in unit tests to figure out 
which may be causing conflict.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-2718-logging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/613.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #613


commit 543990b2a5ab31b6c3335ef19de1ceb022e88365
Author: Rajini Sivaram 
Date:   2015-12-02T08:43:17Z

KAFKA-2718: Add logging to investigate intermittent unit test failures




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2718) Reuse of temporary directories leading to transient unit test failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035477#comment-15035477
 ] 

ASF GitHub Bot commented on KAFKA-2718:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/613

KAFKA-2718: Add logging to investigate intermittent unit test failures

Print port and directories used by zookeeper in unit tests to figure out 
which may be causing conflict.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-2718-logging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/613.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #613


commit 543990b2a5ab31b6c3335ef19de1ceb022e88365
Author: Rajini Sivaram 
Date:   2015-12-02T08:43:17Z

KAFKA-2718: Add logging to investigate intermittent unit test failures




> Reuse of temporary directories leading to transient unit test failures
> --
>
> Key: KAFKA-2718
> URL: https://issues.apache.org/jira/browse/KAFKA-2718
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Affects Versions: 0.9.1.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.1
>
>
> Stack traces in some of the transient unit test failures indicate that 
> temporary directories used for Zookeeper are being reused.
> {quote}
> kafka.common.TopicExistsException: Topic "topic" already exists.
>   at 
> kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:253)
>   at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:237)
>   at kafka.utils.TestUtils$.createTopic(TestUtils.scala:231)
>   at kafka.api.BaseConsumerTest.setUp(BaseConsumerTest.scala:63)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (KAFKA-2718) Reuse of temporary directories leading to transient unit test failures

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram reopened KAFKA-2718:
---

> Reuse of temporary directories leading to transient unit test failures
> --
>
> Key: KAFKA-2718
> URL: https://issues.apache.org/jira/browse/KAFKA-2718
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Affects Versions: 0.9.1.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.1
>
>
> Stack traces in some of the transient unit test failures indicate that 
> temporary directories used for Zookeeper are being reused.
> {quote}
> kafka.common.TopicExistsException: Topic "topic" already exists.
>   at 
> kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:253)
>   at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:237)
>   at kafka.utils.TestUtils$.createTopic(TestUtils.scala:231)
>   at kafka.api.BaseConsumerTest.setUp(BaseConsumerTest.scala:63)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2851) system tests: error copying keytab file

[Anna Povzner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Povzner updated KAFKA-2851:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> system tests: error copying keytab file
> ---
>
> Key: KAFKA-2851
> URL: https://issues.apache.org/jira/browse/KAFKA-2851
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Anna Povzner
>Priority: Minor
>
> It is best to use unique paths for temporary files on the test driver machine 
> so that multiple test jobs don't conflict. 
> If the test driver machine is running multiple ducktape jobs concurrently, as 
> is the case with Confluent nightly test runs, conflicts can occur if the same 
> canonical path is always used.
> In this case, security_config.py copies a file to /tmp/keytab on the test 
> driver machine, while other jobs may remove this from the driver machine. 
> Then you can get errors like this:
> {code}
> 
> test_id:
> 2015-11-17--001.kafkatest.tests.replication_test.ReplicationTest.test_replication_with_broker_failure.security_protocol=SASL_PLAINTEXT.failure_mode=clean_bounce
> status: FAIL
> run time:   1 minute 33.395 seconds
> 
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/tests/runner.py",
>  line 101, in run_all_tests
> result.data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/tests/runner.py",
>  line 151, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 132, in test_replication_with_broker_failure
> self.run_produce_consume_validate(core_test_action=lambda: 
> failures[failure_mode](self))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 66, in run_produce_consume_validate
> core_test_action()
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 132, in 
> self.run_produce_consume_validate(core_test_action=lambda: 
> failures[failure_mode](self))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/tests/replication_test.py",
>  line 43, in clean_bounce
> test.kafka.restart_node(prev_leader_node, clean_shutdown=True)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/kafka/kafka.py",
>  line 275, in restart_node
> self.start_node(node)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/kafka/kafka.py",
>  line 123, in start_node
> self.security_config.setup_node(node)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/tests/kafkatest/services/security/security_config.py",
>  line 130, in setup_node
> node.account.scp_to(MiniKdc.LOCAL_KEYTAB_FILE, SecurityConfig.KEYTAB_PATH)
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/cluster/remoteaccount.py",
>  line 174, in scp_to
> return self._ssh_quiet(self.scp_to_command(src, dest, recursive))
>   File 
> "/var/lib/jenkins/workspace/kafka_system_tests/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.3.8-py2.7.egg/ducktape/cluster/remoteaccount.py",
>  line 219, in _ssh_quiet
> raise e
> CalledProcessError: Command 'scp -o 'HostName 52.33.250.202' -o 'Port 22' -o 
> 'UserKnownHostsFile /dev/null' -o 'StrictHostKeyChecking no' -o 
> 'PasswordAuthentication no' -o 'IdentityFile /var/lib/jenkins/muckrake.pem' 
> -o 'IdentitiesOnly yes' -o 'LogLevel FATAL'  /tmp/keytab 
> ubuntu@worker2:/mnt/security/keytab' returned non-zero exit status 1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2825: Add controller failover to existin...

[apovzner]

GitHub user apovzner opened a pull request:

https://github.com/apache/kafka/pull/618

KAFKA-2825: Add controller failover to existing replication tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apovzner/kafka kafka_2825_01

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/618.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #618


commit fa0b4156d209522b1fe7656f73bb2792d8c932b3
Author: Anna Povzner 
Date:   2015-12-02T22:38:20Z

KAFKA-2825: Add controller failover to existing replication tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2825) Add controller failover to existing replication tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036808#comment-15036808
 ] 

ASF GitHub Bot commented on KAFKA-2825:
---

GitHub user apovzner opened a pull request:

https://github.com/apache/kafka/pull/618

KAFKA-2825: Add controller failover to existing replication tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apovzner/kafka kafka_2825_01

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/618.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #618


commit fa0b4156d209522b1fe7656f73bb2792d8c932b3
Author: Anna Povzner 
Date:   2015-12-02T22:38:20Z

KAFKA-2825: Add controller failover to existing replication tests




> Add controller failover to existing replication tests
> -
>
> Key: KAFKA-2825
> URL: https://issues.apache.org/jira/browse/KAFKA-2825
> Project: Kafka
>  Issue Type: Test
>Reporter: Anna Povzner
>Assignee: Anna Povzner
>
> Extend existing replication tests to include controller failover:
> * clean/hard shutdown
> * clean/hard bounce



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-1342) Slow controlled shutdowns can result in stale shutdown requests

[Onur Karaman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Onur Karaman reassigned KAFKA-1342:
---

Assignee: Onur Karaman  (was: Jiangjie Qin)

> Slow controlled shutdowns can result in stale shutdown requests
> ---
>
> Key: KAFKA-1342
> URL: https://issues.apache.org/jira/browse/KAFKA-1342
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1
>Reporter: Joel Koshy
>Assignee: Onur Karaman
>Priority: Blocker
>  Labels: newbie++, newbiee
> Fix For: 0.10.0.0
>
>
> I don't think this is a bug introduced in 0.8.1., but triggered by the fact
> that controlled shutdown seems to have become slower in 0.8.1 (will file a
> separate ticket to investigate that). When doing a rolling bounce, it is
> possible for a bounced broker to stop all its replica fetchers since the
> previous PID's shutdown requests are still being shutdown.
> - 515 is the controller
> - Controlled shutdown initiated for 503
> - Controller starts controlled shutdown for 503
> - The controlled shutdown takes a long time in moving leaders and moving
>   follower replicas on 503 to the offline state.
> - So 503's read from the shutdown channel times out and a new channel is
>   created. It issues another shutdown request.  This request (since it is a
>   new channel) is accepted at the controller's socket server but then waits
>   on the broker shutdown lock held by the previous controlled shutdown which
>   is still in progress.
> - The above step repeats for the remaining retries (six more requests).
> - 503 hits SocketTimeout exception on reading the response of the last
>   shutdown request and proceeds to do an unclean shutdown.
> - The controller's onBrokerFailure call-back fires and moves 503's replicas
>   to offline (not too important in this sequence).
> - 503 is brought back up.
> - The controller's onBrokerStartup call-back fires and moves its replicas
>   (and partitions) to online state. 503 starts its replica fetchers.
> - Unfortunately, the (phantom) shutdown requests are still being handled and
>   the controller sends StopReplica requests to 503.
> - The first shutdown request finally finishes (after 76 minutes in my case!).
> - The remaining shutdown requests also execute and do the same thing (sends
>   StopReplica requests for all partitions to
>   503).
> - The remaining requests complete quickly because they end up not having to
>   touch zookeeper paths - no leaders left on the broker and no need to
>   shrink ISR in zookeeper since it has already been done by the first
>   shutdown request.
> - So in the end-state 503 is up, but effectively idle due to the previous
>   PID's shutdown requests.
> There are some obvious fixes that can be made to controlled shutdown to help
> address the above issue. E.g., we don't really need to move follower
> partitions to Offline. We did that as an "optimization" so the broker falls
> out of ISR sooner - which is helpful when producers set required.acks to -1.
> However it adds a lot of latency to controlled shutdown. Also, (more
> importantly) we should have a mechanism to abort any stale shutdown process.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2931) Consumer rolling upgrade test case

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036817#comment-15036817
 ] 

ASF GitHub Bot commented on KAFKA-2931:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/619

KAFKA-2931: add system test for consumer rolling upgrades



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-2931

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/619.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #619


commit 41e4ac43ed008b1043292cfc879992de3b5098ac
Author: Jason Gustafson 
Date:   2015-12-02T22:48:14Z

KAFKA-2931: add system test for consumer rolling upgrades




> Consumer rolling upgrade test case
> --
>
> Key: KAFKA-2931
> URL: https://issues.apache.org/jira/browse/KAFKA-2931
> Project: Kafka
>  Issue Type: Test
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>
> We need a system test which covers the rolling upgrade process for the new 
> consumer. The idea is to start the consumers with a "range" assignment 
> strategy and then upgrade to "round-robin" without any down-time. This 
> validates the coordinator's protocol selection process.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2931: add system test for consumer rolli...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/619

KAFKA-2931: add system test for consumer rolling upgrades



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-2931

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/619.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #619


commit 41e4ac43ed008b1043292cfc879992de3b5098ac
Author: Jason Gustafson 
Date:   2015-12-02T22:48:14Z

KAFKA-2931: add system test for consumer rolling upgrades




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk8 #199

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-2851 Using random file names for local kdc files to avoid

[wangguoz] KAFKA-2880: consumer should handle disconnect/timeout for metadata

--
[...truncated 6799 lines...]
org.apache.kafka.connect.json.JsonConverterTest > timeToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > structToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
testConnectSchemaMetadataTranslation PASSED

org.apache.kafka.connect.json.JsonConverterTest > shortToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > dateToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > doubleToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > timeToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToConnectStringKeys PASSED

org.apache.kafka.connect.json.JsonConverterTest > floatToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > decimalToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > arrayToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
testCacheSchemaToConnectConversion PASSED

org.apache.kafka.connect.json.JsonConverterTest > booleanToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > bytesToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > doubleToConnect PASSED
:connect:runtime:checkstyleMain
:connect:runtime:compileTestJavawarning: [options] bootstrap class path not set 
in conjunction with -source 1.7
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 warning

:connect:runtime:processTestResources
:connect:runtime:testClasses
:connect:runtime:checkstyleTest
:connect:runtime:test

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testCommit PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testCommitTaskFlushFailure PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testCommitTaskSuccessAndFlushFailure PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testCommitConsumerFailure PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testCommitTimeout 
PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testAssignmentPauseResume PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testRewind PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testPollsInBackground PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskTest > testPollRedelivery PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testCreateSourceConnector PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testCreateAndStop PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testCreateConnectorAlreadyExists PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testCreateSinkConnector PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testAccessors PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testPutTaskConfigs PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testDestroyConnector PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testNormalJoinGroupFollower PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testMetadata PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testLeaderPerformAssignment1 PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testLeaderPerformAssignment2 PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testJoinLeaderCannotAssign PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testRejoinGroup PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testNormalJoinGroupLeader PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnectorAlreadyExists PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testAccessors PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testDestroyConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinAssignment PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testHaltCleansUpWorker PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigAdded PASSED

[jira] [Created] (KAFKA-2934) Offset storage file configuration in Connect standalone mode is not included in StandaloneConfig

[Ewen Cheslack-Postava (JIRA)]

Ewen Cheslack-Postava created KAFKA-2934:


 Summary: Offset storage file configuration in Connect standalone 
mode is not included in StandaloneConfig
 Key: KAFKA-2934
 URL: https://issues.apache.org/jira/browse/KAFKA-2934
 Project: Kafka
  Issue Type: Bug
  Components: copycat
Affects Versions: 0.9.0.0
Reporter: Ewen Cheslack-Postava
Assignee: Ewen Cheslack-Postava


The config is coded directly in FileOffsetBackingStore rather than being listed 
(and validated) in StandaloneConfig. This also means it wouldn't be included if 
we autogenerated docs from the config classes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2939) Make AbstractConfig.logUnused() tunable for clients

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036977#comment-15036977
 ] 

Jay Kreps commented on KAFKA-2939:
--

That was the whole point of the original implementation. Looking at it now some 
weird stuff has been done. But the idea was that you have a config class that 
tracks access to the configs and that way even if there are custom configs 
defined in a serializer their use should get recorded and they shouldn't get 
logged as unused.

> Make AbstractConfig.logUnused() tunable for clients
> ---
>
> Key: KAFKA-2939
> URL: https://issues.apache.org/jira/browse/KAFKA-2939
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.1.0
>
>
> Today we always log unused configs in KafkaProducer / KafkaConsumer in their 
> constructors, however for some cases like Kafka Streams that make use of 
> these clients, other configs may be passed in to configure Partitioner / 
> Serializer classes, etc. So it would be better to make this function call 
> optional to avoid printing unnecessary and confusing WARN entries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2804: manage changelog topics through ZK...

[guozhangwang]

GitHub user guozhangwang reopened a pull request:

https://github.com/apache/kafka/pull/579

KAFKA-2804: manage changelog topics through ZK in PartitionAssignor



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka K2804

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/579.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #579


commit b2aad7cb73431b923170ea3cc2dd193c4f10
Author: Guozhang Wang 
Date:   2015-11-22T02:42:49Z

comment links

commit d35a1599718b831deefcb47d5d11a3e59b0c31a1
Author: wangg...@gmail.com 
Date:   2015-11-22T02:51:08Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
K2804

commit d23db8fd8d7810dfcf7b1be2daa25cd074127eb4
Author: Guozhang Wang 
Date:   2015-11-23T23:23:16Z

auto create topic in partition assignor and block wait on topic partition 
existence

commit cf263fdd23eaa268c29f94cd0c1ac9455add9a0f
Author: Guozhang Wang 
Date:   2015-11-24T00:10:08Z

fix unit tests

commit dd571904bd1bb834215c51806ee1a23d6b082670
Author: Guozhang Wang 
Date:   2015-11-24T00:10:23Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
K2804

commit 3f5c1c34cc52c93758b58c7ad6f018402e06d31c
Author: Guozhang Wang 
Date:   2015-12-02T00:02:32Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
K2804

commit 67ef0321ae7398114c0c6bd5b71df263cfe2f4bc
Author: Guozhang Wang 
Date:   2015-12-02T00:22:11Z

incoporate comments

commit 0e5c7c3be8db56366895f833a0092ca177fdbda5
Author: Guozhang Wang 
Date:   2015-12-02T01:37:53Z

refactor PartitionGrouper

commit f76ee8b94da66104b21534cf1c75c9314d995acc
Author: Guozhang Wang 
Date:   2015-12-03T00:58:07Z

add Job-Id into StreamingConfig

commit 0aa06ed3e992f4dd7ea7aa72707a8f53f5b52d67
Author: Guozhang Wang 
Date:   2015-12-03T00:58:15Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
K2804

commit 1c9827a62dcdc8206bd1309b6bc474c68bf56952
Author: Guozhang Wang 
Date:   2015-12-03T02:31:58Z

some minor fixes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: Kafka 2902 streaming config use get base consu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/596


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2938) Socket server may throw IllegalStateException

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037259#comment-15037259
 ] 

Jiangjie Qin commented on KAFKA-2938:
-

[~junrao]You are right. We actually mute the channel after finish one receive 
and unmute it after a send finishes. I will close the ticket.

> Socket server may throw IllegalStateException
> -
>
> Key: KAFKA-2938
> URL: https://issues.apache.org/jira/browse/KAFKA-2938
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> The Processor in SocketServer will try to drain all the responses in the 
> response queue and pass them to selector for sending. 
> If there are more than one response in the response queue (e.g. a producer 
> with max in flight request greater than 1), an IllegalStateException would be 
> thrown because the processor will call selector.send() before the previous 
> send is finished.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2804) Create / Update changelog topics upon state store initialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037120#comment-15037120
 ] 

ASF GitHub Bot commented on KAFKA-2804:
---

GitHub user guozhangwang reopened a pull request:

https://github.com/apache/kafka/pull/579

KAFKA-2804: manage changelog topics through ZK in PartitionAssignor



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka K2804

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/579.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #579


commit b2aad7cb73431b923170ea3cc2dd193c4f10
Author: Guozhang Wang 
Date:   2015-11-22T02:42:49Z

comment links

commit d35a1599718b831deefcb47d5d11a3e59b0c31a1
Author: wangg...@gmail.com 
Date:   2015-11-22T02:51:08Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
K2804

commit d23db8fd8d7810dfcf7b1be2daa25cd074127eb4
Author: Guozhang Wang 
Date:   2015-11-23T23:23:16Z

auto create topic in partition assignor and block wait on topic partition 
existence

commit cf263fdd23eaa268c29f94cd0c1ac9455add9a0f
Author: Guozhang Wang 
Date:   2015-11-24T00:10:08Z

fix unit tests

commit dd571904bd1bb834215c51806ee1a23d6b082670
Author: Guozhang Wang 
Date:   2015-11-24T00:10:23Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
K2804

commit 3f5c1c34cc52c93758b58c7ad6f018402e06d31c
Author: Guozhang Wang 
Date:   2015-12-02T00:02:32Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
K2804

commit 67ef0321ae7398114c0c6bd5b71df263cfe2f4bc
Author: Guozhang Wang 
Date:   2015-12-02T00:22:11Z

incoporate comments

commit 0e5c7c3be8db56366895f833a0092ca177fdbda5
Author: Guozhang Wang 
Date:   2015-12-02T01:37:53Z

refactor PartitionGrouper

commit f76ee8b94da66104b21534cf1c75c9314d995acc
Author: Guozhang Wang 
Date:   2015-12-03T00:58:07Z

add Job-Id into StreamingConfig

commit 0aa06ed3e992f4dd7ea7aa72707a8f53f5b52d67
Author: Guozhang Wang 
Date:   2015-12-03T00:58:15Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
K2804

commit 1c9827a62dcdc8206bd1309b6bc474c68bf56952
Author: Guozhang Wang 
Date:   2015-12-03T02:31:58Z

some minor fixes




> Create / Update changelog topics upon state store initialization
> 
>
> Key: KAFKA-2804
> URL: https://issues.apache.org/jira/browse/KAFKA-2804
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>
> When state store instances that are logging-backed are initialized, we need 
> to check if the corresponding change log topics have been created with the 
> right number of partitions:
> 1) If not exist, create topic
> 2) If expected #.partitions < actual #.partitions, delete and re-create topic.
> 3) If expected #.partitions > actual #.partitions, add partitions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2804: manage changelog topics through ZK...

[guozhangwang]

Github user guozhangwang closed the pull request at:

https://github.com/apache/kafka/pull/579


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2938) Socket server may throw IllegalStateException

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037137#comment-15037137
 ] 

Jun Rao commented on KAFKA-2938:


[~becket_qin], can this actually happen? In SocketServer, we only process one 
request at a time from a given client. So before we completely send the 
response for a request, socket server won't take another request from the same 
socket. So, there should never be two outstanding responses for the same 
client. The producer can send multiple requests in flight, but they will just 
be queued up in the socket buffer on the broker.

> Socket server may throw IllegalStateException
> -
>
> Key: KAFKA-2938
> URL: https://issues.apache.org/jira/browse/KAFKA-2938
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> The Processor in SocketServer will try to drain all the responses in the 
> response queue and pass them to selector for sending. 
> If there are more than one response in the response queue (e.g. a producer 
> with max in flight request greater than 1), an IllegalStateException would be 
> thrown because the processor will call selector.send() before the previous 
> send is finished.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #871

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-2902: streaming config use get base consumer configs.

--
[...truncated 2808 lines...]
kafka.log.LogTest > testTimeBasedLogRollJitter PASSED

kafka.log.LogTest > testParseTopicPartitionName PASSED

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.log.OffsetIndexTest > lookupExtremeCases PASSED

kafka.log.OffsetIndexTest > appendTooMany PASSED

kafka.log.OffsetIndexTest > randomLookupTest PASSED

kafka.log.OffsetIndexTest > testReopen PASSED

kafka.log.OffsetIndexTest > appendOutOfOrder PASSED

kafka.log.OffsetIndexTest > truncate PASSED

kafka.log.LogSegmentTest > testRecoveryWithCorruptMessage PASSED

kafka.log.LogSegmentTest > testRecoveryFixesCorruptIndex PASSED

kafka.log.LogSegmentTest > testReadFromGap PASSED

kafka.log.LogSegmentTest > testTruncate PASSED

kafka.log.LogSegmentTest > testReadBeforeFirstOffset PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeAppendMessage PASSED

kafka.log.LogSegmentTest > testChangeFileSuffixes PASSED

kafka.log.LogSegmentTest > testMaxOffset PASSED

kafka.log.LogSegmentTest > testNextOffsetCalculation PASSED

kafka.log.LogSegmentTest > testReadOnEmptySegment PASSED

kafka.log.LogSegmentTest > testReadAfterLast PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeClearShutdown PASSED

kafka.log.LogSegmentTest > testTruncateFull PASSED

kafka.log.CleanerTest > testBuildOffsetMap PASSED

kafka.log.CleanerTest > testSegmentGrouping PASSED

kafka.log.CleanerTest > testCleanSegmentsWithAbort PASSED

kafka.log.CleanerTest > testSegmentGroupingWithSparseOffsets PASSED

kafka.log.CleanerTest > testRecoveryAfterCrash PASSED

kafka.log.CleanerTest > testLogToClean PASSED

kafka.log.CleanerTest > testCleaningWithDeletes PASSED

kafka.log.CleanerTest > testCleanSegments PASSED

kafka.log.CleanerTest > testCleaningWithUnkeyedMessages PASSED

kafka.log.OffsetMapTest > testClear PASSED

kafka.log.OffsetMapTest > testBasicValidation PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupStable PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatIllegalGeneration 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testDescribeGroupWrongCoordinator PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupRebalancing 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaderFailureInSyncGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testGenerationIdIncrementsOnRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFromIllegalGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testInvalidGroupId PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesStableGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatDuringRebalanceCausesRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentGroupProtocol PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooLarge PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooSmall PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupEmptyAssignment 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetWithDefaultGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupFromUnchangedLeaderShouldRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesRebalancingGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFollowerAfterLeader PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testCommitOffsetInAwaitingSync 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testJoinGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupFromUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentProtocolType PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetFromUnknownGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testLeaveGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerNewGroup PASSED

[jira] [Commented] (KAFKA-2942) Inadvertent auto-commit when pre-fetching can cause message loss

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037341#comment-15037341
 ] 

ASF GitHub Bot commented on KAFKA-2942:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/623

KAFKA-2942: inadvertent auto-commit when pre-fetching can cause message loss



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-2942

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/623.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #623


commit 01872110576a82f851e791422f5fec3f797711e7
Author: Jason Gustafson 
Date:   2015-12-03T06:18:01Z

KAFKA-2942: inadvertent auto-commit when pre-fetching can cause message loss




> Inadvertent auto-commit when pre-fetching can cause message loss
> 
>
> Key: KAFKA-2942
> URL: https://issues.apache.org/jira/browse/KAFKA-2942
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>
> Before returning from KafkaConsumer.poll(), we update the consumed position 
> and invoke poll(0) to send new fetches. In doing so, it is possible that an 
> auto-commit is triggered which would commit the updated offsets which hasn't 
> yet been returned. If the process then crashes before consuming the messages, 
> there would be a gap in the delivery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2942) Inadvertent auto-commit when pre-fetching can cause message loss

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-2942:
--

 Summary: Inadvertent auto-commit when pre-fetching can cause 
message loss
 Key: KAFKA-2942
 URL: https://issues.apache.org/jira/browse/KAFKA-2942
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson
Assignee: Jason Gustafson


Before returning from KafkaConsumer.poll(), we update the consumed position and 
invoke poll(0) to send new fetches. In doing so, it is possible that an 
auto-commit is triggered which would commit the updated offsets which hasn't 
yet been returned. If the process then crashes before consuming the messages, 
there would be a gap in the delivery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2942: inadvertent auto-commit when pre-f...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/623

KAFKA-2942: inadvertent auto-commit when pre-fetching can cause message loss



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-2942

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/623.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #623


commit 01872110576a82f851e791422f5fec3f797711e7
Author: Jason Gustafson 
Date:   2015-12-03T06:18:01Z

KAFKA-2942: inadvertent auto-commit when pre-fetching can cause message loss




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2902) StreamingConfig getConsumerConfiigs uses getRestoreConsumerConfigs instead of getBaseConsumerConfigs

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2902:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 596
[https://github.com/apache/kafka/pull/596]

> StreamingConfig getConsumerConfiigs uses getRestoreConsumerConfigs instead of 
>  getBaseConsumerConfigs
> -
>
> Key: KAFKA-2902
> URL: https://issues.apache.org/jira/browse/KAFKA-2902
> Project: Kafka
>  Issue Type: Bug
>  Components: kafka streams
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
> Fix For: 0.9.1.0
>
>
> When starting a KafkaStreaming instance the 
> StreamingConfig.getConsumerConfigs method uses the getRestoreConsumerConfigs 
> to retrieve properties. But this method removes the groupId property which 
> causes an error and the KafkaStreaming instance shuts down.  On 
> KafkaStreaming startup StreamingConfig should use getBaseConsumerConfigs 
> instead.
> Exception in thread "StreamThread-1" org.apache.kafka.common.KafkaException: 
> org.apache.kafka.common.errors.ApiException: The configured groupId is invalid
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:309)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:198)
> Caused by: org.apache.kafka.common.errors.ApiException: The configured 
> groupId is invalid 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2804) Create / Update changelog topics upon state store initialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037119#comment-15037119
 ] 

ASF GitHub Bot commented on KAFKA-2804:
---

Github user guozhangwang closed the pull request at:

https://github.com/apache/kafka/pull/579


> Create / Update changelog topics upon state store initialization
> 
>
> Key: KAFKA-2804
> URL: https://issues.apache.org/jira/browse/KAFKA-2804
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>
> When state store instances that are logging-backed are initialized, we need 
> to check if the corresponding change log topics have been created with the 
> right number of partitions:
> 1) If not exist, create topic
> 2) If expected #.partitions < actual #.partitions, delete and re-create topic.
> 3) If expected #.partitions > actual #.partitions, add partitions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2940) Make available to use any Java options at startup scripts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037136#comment-15037136
 ] 

ASF GitHub Bot commented on KAFKA-2940:
---

GitHub user sasakitoa opened a pull request:

https://github.com/apache/kafka/pull/621

KAFKA-2940: Make available to use any Java options at startup scripts

We cannot specify any Java options (e.g. option for remote debugging) at 
startup scrips such as kafka-server-start.sh .
This ticket makes we can specify to use "JAVA_OPTS" environmental variables.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sasakitoa/kafka java_opt

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/621.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #621


commit aaf58f1de02b56468911e416fd3053d2d4acf320
Author: Sasaki Toru 
Date:   2015-12-03T02:39:08Z

Add JAVA_OPTS to specify any Java options to use start scripts.

commit 06f688e386b5a12301e0980fda794b21248b0d64
Author: Sasaki Toru 
Date:   2015-12-03T02:41:15Z

Merge branch 'trunk' of https://github.com/apache/kafka into java_opt




> Make available to use any Java options at startup scripts
> -
>
> Key: KAFKA-2940
> URL: https://issues.apache.org/jira/browse/KAFKA-2940
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Sasaki Toru
>Priority: Minor
> Fix For: 0.9.0.1
>
>
> We cannot specify any Java options (e.g. option for remote debugging) at 
> startup scrips such as kafka-server-start.sh .
> This ticket makes we can specify to use "JAVA_OPTS" environmental variables.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2940: Make available to use any Java opt...

[sasakitoa]

GitHub user sasakitoa opened a pull request:

https://github.com/apache/kafka/pull/621

KAFKA-2940: Make available to use any Java options at startup scripts

We cannot specify any Java options (e.g. option for remote debugging) at 
startup scrips such as kafka-server-start.sh .
This ticket makes we can specify to use "JAVA_OPTS" environmental variables.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sasakitoa/kafka java_opt

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/621.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #621


commit aaf58f1de02b56468911e416fd3053d2d4acf320
Author: Sasaki Toru 
Date:   2015-12-03T02:39:08Z

Add JAVA_OPTS to specify any Java options to use start scripts.

commit 06f688e386b5a12301e0980fda794b21248b0d64
Author: Sasaki Toru 
Date:   2015-12-03T02:41:15Z

Merge branch 'trunk' of https://github.com/apache/kafka into java_opt




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2308) New producer + Snappy face un-compression errors after broker restart

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037210#comment-15037210
 ] 

ASF GitHub Bot commented on KAFKA-2308:
---

Github user darionyaphet commented on the pull request:

https://github.com/apache/storm/pull/801#issuecomment-161509467
  
Hi @knusbaum @revans2 I read `Kafka Release Notes Version 0.8.2.2`  and 
found a bug fixed 
([KAFKA-2308](https://issues.apache.org/jira/browse/KAFKA-2308)) about New 
producer and Snappy un-compression errors when Kafka Broker restart . So I 
think this is maybe useful .


> New producer + Snappy face un-compression errors after broker restart
> -
>
> Key: KAFKA-2308
> URL: https://issues.apache.org/jira/browse/KAFKA-2308
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
> Fix For: 0.9.0.0, 0.8.2.2
>
> Attachments: KAFKA-2308.patch
>
>
> Looks like the new producer, when used with Snappy, following a broker 
> restart is sending messages the brokers can't decompress. This issue was 
> discussed at few mailing lists thread, but I don't think we ever resolved it.
> I can reproduce with trunk and Snappy 1.1.1.7. 
> To reproduce:
> 1. Start 3 brokers
> 2. Create a topic with 3 partitions and 3 replicas each.
> 2. Start performance producer with --new-producer --compression-codec 2 (and 
> set the number of messages to fairly high, to give you time. I went with 10M)
> 3. Bounce one of the brokers
> 4. The log of one of the surviving nodes should contain errors like:
> {code}
> 2015-07-02 13:45:59,300 ERROR kafka.server.ReplicaManager: [Replica Manager 
> on Broker 66]: Error processing append operation on partition [t3,0]
> kafka.common.KafkaException:
> at 
> kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:94)
> at 
> kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:64)
> at 
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
> at 
> kafka.message.ByteBufferMessageSet$$anon$2.innerDone(ByteBufferMessageSet.scala:177)
> at 
> kafka.message.ByteBufferMessageSet$$anon$2.makeNext(ByteBufferMessageSet.scala:218)
> at 
> kafka.message.ByteBufferMessageSet$$anon$2.makeNext(ByteBufferMessageSet.scala:173)
> at 
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at 
> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
> at 
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
> at 
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
> at 
> scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
> at scala.collection.AbstractIterator.to(Iterator.scala:1157)
> at 
> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
> at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
> at 
> kafka.message.ByteBufferMessageSet.validateMessagesAndAssignOffsets(ByteBufferMessageSet.scala:267)
> at kafka.log.Log.liftedTree1$1(Log.scala:327)
> at kafka.log.Log.append(Log.scala:326)
> at 
> kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:423)
> at 
> kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:409)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
> at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:268)
> at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:409)
> at 
> kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:365)
> at 
> kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:350)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at 

Build failed in Jenkins: kafka-trunk-jdk8 #200

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-2902: streaming config use get base consumer configs.

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-2 (docker Ubuntu ubuntu) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision a5382a333ebb51a10c1a1cab46d66f10abff128a 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a5382a333ebb51a10c1a1cab46d66f10abff128a
 > git rev-list 9fb1e25738f89b75a36ef69f730b0e138ccd55b1 # timeout=10
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson8833289141292977000.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 13.907 secs
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson6476738416916861616.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.9/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:clean UP-TO-DATE
:clients:clean
:connect:clean UP-TO-DATE
:core:clean UP-TO-DATE
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJava
Download https://repo1.maven.org/maven2/net/jpountz/lz4/lz4/1.3/lz4-1.3.pom
Download https://repo1.maven.org/maven2/net/jpountz/lz4/lz4/1.3/lz4-1.3.jar
:jar_core_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture snapshot of input files for task 'compileJava' during 
up-to-date check.  See stacktrace for details.
> Could not add entry 
> '/home/jenkins/.gradle/caches/modules-2/files-2.1/net.jpountz.lz4/lz4/1.3/792d5e592f6f3f0c1a3337cd0ac84309b544f8f4/lz4-1.3.jar'
>  to cache fileHashes.bin 
> (

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 15.207 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
ERROR: Publisher 'Publish JUnit test result report' failed: No test report 
files were found. Configuration error?
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2

[GitHub] kafka pull request: KAFKA-2924: support offsets topic in DumpLogSe...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/622

KAFKA-2924: support offsets topic in DumpLogSegments



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-2924

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/622.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #622


commit 2b2bb8af1f2e46ec8a36f23e297b2aeea497f648
Author: Jason Gustafson 
Date:   2015-12-03T01:56:16Z

KAFKA-2924: support offsets topic in DumpLogSegments




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2924) Add offsets/group metadata decoder so that DumpLogSegments can be used with the offsets topic

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037230#comment-15037230
 ] 

ASF GitHub Bot commented on KAFKA-2924:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/622

KAFKA-2924: support offsets topic in DumpLogSegments



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-2924

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/622.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #622


commit 2b2bb8af1f2e46ec8a36f23e297b2aeea497f648
Author: Jason Gustafson 
Date:   2015-12-03T01:56:16Z

KAFKA-2924: support offsets topic in DumpLogSegments




> Add offsets/group metadata decoder so that DumpLogSegments can be used with 
> the offsets topic
> -
>
> Key: KAFKA-2924
> URL: https://issues.apache.org/jira/browse/KAFKA-2924
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>
> We've only implemented a MessageFormatter for use with the ConsoleConsumer, 
> but it would be helpful to be able to pull offsets/metadata from log files 
> directly in testing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2938) Socket server may throw IllegalStateException

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin resolved KAFKA-2938.
-
Resolution: Not A Problem

> Socket server may throw IllegalStateException
> -
>
> Key: KAFKA-2938
> URL: https://issues.apache.org/jira/browse/KAFKA-2938
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> The Processor in SocketServer will try to drain all the responses in the 
> response queue and pass them to selector for sending. 
> If there are more than one response in the response queue (e.g. a producer 
> with max in flight request greater than 1), an IllegalStateException would be 
> thrown because the processor will call selector.send() before the previous 
> send is finished.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2941) Docs for key/value converter in Kafka connect are unclear

[Ewen Cheslack-Postava (JIRA)]

Ewen Cheslack-Postava created KAFKA-2941:


 Summary: Docs for key/value converter in Kafka connect are unclear
 Key: KAFKA-2941
 URL: https://issues.apache.org/jira/browse/KAFKA-2941
 Project: Kafka
  Issue Type: Bug
  Components: copycat
Affects Versions: 0.9.0.0
Reporter: Ewen Cheslack-Postava
Assignee: Ewen Cheslack-Postava
Priority: Minor


These docs don't really explain what the configs do or why users might want to 
change them.

Via [~gwenshap], something like this would be better: "Converter class for key 
Connect data. This controls the format of the data that will be written either 
to Kafka or to a sink system. Popular formats include Json and Avro"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2937) Topics marked for delete in Zookeeper may become undeletable

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-2937:
-

 Summary: Topics marked for delete in Zookeeper may become 
undeletable
 Key: KAFKA-2937
 URL: https://issues.apache.org/jira/browse/KAFKA-2937
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.9.0.0
Reporter: Rajini Sivaram


In our clusters, we occasionally see topics marked for delete, but never 
actually deleted. It may be due to brokers being restarted while tests were 
running, but further restarts of Kafka dont fix the problem. The topics remain 
marked for delete in Zookeeper.

Topic describe shows:
{quote}
Topic:testtopic PartitionCount:1ReplicationFactor:3 Configs:
Topic: testtopicPartition: 0Leader: noneReplicas: 3,4,0 
Isr: 
{quote}

Kafka logs show:
{quote}
2015-12-02 15:53:30,152] ERROR Controller 2 epoch 213 initiated state change of 
replica 3 for partition [testtopic,0] from OnlineReplica to OfflineReplica 
failed (state.change.logger)
kafka.common.StateChangeFailedException: Failed to change state of replica 3 
for partition [testtopic,0] since the leader and isr path in zookeeper is empty
at 
kafka.controller.ReplicaStateMachine.handleStateChange(ReplicaStateMachine.scala:269)
at 
kafka.controller.ReplicaStateMachine$$anonfun$handleStateChanges$2.apply(ReplicaStateMachine.scala:114)
at 
kafka.controller.ReplicaStateMachine$$anonfun$handleStateChanges$2.apply(ReplicaStateMachine.scala:114)
at 
scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:322)
at 
scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:978)
at 
kafka.controller.ReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:114)
at 
kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2.apply(TopicDeletionManager.scala:342)
at 
kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2.apply(TopicDeletionManager.scala:334)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
at 
kafka.controller.TopicDeletionManager.startReplicaDeletion(TopicDeletionManager.scala:334)
at 
kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$onPartitionDeletion(TopicDeletionManager.scala:367)
at 
kafka.controller.TopicDeletionManager$$anonfun$kafka$controller$TopicDeletionManager$$onTopicDeletion$2.apply(TopicDeletionManager.scala:313)
at 
kafka.controller.TopicDeletionManager$$anonfun$kafka$controller$TopicDeletionManager$$onTopicDeletion$2.apply(TopicDeletionManager.scala:312)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
at 
kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$onTopicDeletion(TopicDeletionManager.scala:312)
at 
kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1$$anonfun$apply$mcV$sp$4.apply(TopicDeletionManager.scala:431)
at 
kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1$$anonfun$apply$mcV$sp$4.apply(TopicDeletionManager.scala:403)
at scala.collection.immutable.Set$Set2.foreach(Set.scala:111)
at 
kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply$mcV$sp(TopicDeletionManager.scala:403)
at 
kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply(TopicDeletionManager.scala:397)
at 
kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply(TopicDeletionManager.scala:397)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
at 
kafka.controller.TopicDeletionManager$DeleteTopicsThread.doWork(TopicDeletionManager.scala:397)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
{quote}  
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2936) Socket server selector can stuck on one send in tight loop.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-2936:

Component/s: network

> Socket server selector can stuck on one send in tight loop.
> ---
>
> Key: KAFKA-2936
> URL: https://issues.apache.org/jira/browse/KAFKA-2936
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> When broker was sending a FetchResponse it is possible that the data to be 
> sent back is truncated. In this case, a KafkaException will be thrown. This 
> exception is caught by processor and the selector will be sending the message 
> in a tight while loop. It will continue doing this until the socket is closed 
> by the client due to request timeout.
> We should have a max retry for the response send.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2936) Socket server selector can stuck on one send in tight loop.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-2936:

Assignee: Onur Karaman

> Socket server selector can stuck on one send in tight loop.
> ---
>
> Key: KAFKA-2936
> URL: https://issues.apache.org/jira/browse/KAFKA-2936
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Onur Karaman
> Fix For: 0.9.0.1
>
>
> When broker was sending a FetchResponse it is possible that the data to be 
> sent back is truncated. In this case, a KafkaException will be thrown. This 
> exception is caught by processor and the selector will be sending the message 
> in a tight while loop. It will continue doing this until the socket is closed 
> by the client due to request timeout.
> We should have a max retry for the response send.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Manually ported changes in 8c3c9548b636...

[granders]

GitHub user granders opened a pull request:

https://github.com/apache/kafka/pull/620

MINOR: Manually ported changes in 8c3c9548b636cdf760d2537afe115942d13bc003 

Porting manually to 0.9.0 (no cherry-pick due to conflicts)

This will allow concurrent system test runs on the same machine

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka 
minor-0.9.0-minikdc-conflict

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/620.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #620


commit ee8a7ce9a4789e4538cf9d337b2fc0d80204da06
Author: Geoff Anderson 
Date:   2015-12-02T23:42:57Z

Manually ported changes in 8c3c9548b636cdf760d2537afe115942d13bc003 to 
0.9.0 (no cherry-pick due to conflicts)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-2936) Socket server selector can stuck on one send in tight loop.

[Jiangjie Qin (JIRA)]

Jiangjie Qin created KAFKA-2936:
---

 Summary: Socket server selector can stuck on one send in tight 
loop.
 Key: KAFKA-2936
 URL: https://issues.apache.org/jira/browse/KAFKA-2936
 Project: Kafka
  Issue Type: Bug
Reporter: Jiangjie Qin


When broker was sending a FetchResponse it is possible that the data to be sent 
back is truncated. In this case, a KafkaException will be thrown. This 
exception is caught by processor and the selector will be sending the message 
in a tight while loop. It will continue doing this until the socket is closed 
by the client due to request timeout.

We should have a max retry for the response send.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2936) Socket server selector can stuck on one send in tight loop.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-2936:

Affects Version/s: 0.9.0.0

> Socket server selector can stuck on one send in tight loop.
> ---
>
> Key: KAFKA-2936
> URL: https://issues.apache.org/jira/browse/KAFKA-2936
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> When broker was sending a FetchResponse it is possible that the data to be 
> sent back is truncated. In this case, a KafkaException will be thrown. This 
> exception is caught by processor and the selector will be sending the message 
> in a tight while loop. It will continue doing this until the socket is closed 
> by the client due to request timeout.
> We should have a max retry for the response send.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2936) Socket server selector can stuck on one send in tight loop.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-2936:

Fix Version/s: 0.9.0.1

> Socket server selector can stuck on one send in tight loop.
> ---
>
> Key: KAFKA-2936
> URL: https://issues.apache.org/jira/browse/KAFKA-2936
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> When broker was sending a FetchResponse it is possible that the data to be 
> sent back is truncated. In this case, a KafkaException will be thrown. This 
> exception is caught by processor and the selector will be sending the message 
> in a tight while loop. It will continue doing this until the socket is closed 
> by the client due to request timeout.
> We should have a max retry for the response send.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2938) Socket server may throw IllegalStateException

[Jiangjie Qin (JIRA)]

Jiangjie Qin created KAFKA-2938:
---

 Summary: Socket server may throw IllegalStateException
 Key: KAFKA-2938
 URL: https://issues.apache.org/jira/browse/KAFKA-2938
 Project: Kafka
  Issue Type: Bug
  Components: network
Affects Versions: 0.9.0.0
Reporter: Jiangjie Qin
Assignee: Jiangjie Qin
 Fix For: 0.9.0.1


The Processor in SocketServer will try to drain all the responses in the 
response queue and pass them to selector for sending. 

If there are more than one response in the response queue (e.g. a producer with 
max in flight request greater than 1), an IllegalStateException would be thrown 
because the processor will call selector.send() before the previous send is 
finished.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Kafka Summit Registration and CFP

[Jay Kreps]

Hey Everyone,

As you may have heard, Confluent is hosting the first ever Kafka
Summit. It'll be in San Francisco on Tuesday, April 26, 2016.

We'll be announcing open registration tomorrow, but I wanted to let
everyone here know first, and also let you know there is a $50
community discount. To get the discount enter this promotional code:
COMMUNITY-KS2016-50D

Don't put the discount code on twitter, it's just for the people on
the mailing list :-)

Also, the call for proposals is open--we'd love to have you give a
talk on how you're using Kafka, your experience with stream processing
frameworks and applications, or anything else in the streaming data
space. Submission deadline is January 11.

Hope to see you all there!

http://www.kafka-summit.org

Cheers,

-Jay

[jira] [Commented] (KAFKA-2934) Offset storage file configuration in Connect standalone mode is not included in StandaloneConfig

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036882#comment-15036882
 ] 

Ewen Cheslack-Postava commented on KAFKA-2934:
--

Also found a couple of other configs that should be in the config class: 
offset.storage.topic and config.storage.topic.

> Offset storage file configuration in Connect standalone mode is not included 
> in StandaloneConfig
> 
>
> Key: KAFKA-2934
> URL: https://issues.apache.org/jira/browse/KAFKA-2934
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Affects Versions: 0.9.0.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>
> The config is coded directly in FileOffsetBackingStore rather than being 
> listed (and validated) in StandaloneConfig. This also means it wouldn't be 
> included if we autogenerated docs from the config classes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2937) Topics marked for delete in Zookeeper may become undeletable

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat reassigned KAFKA-2937:
--

Assignee: Mayuresh Gharat

> Topics marked for delete in Zookeeper may become undeletable
> 
>
> Key: KAFKA-2937
> URL: https://issues.apache.org/jira/browse/KAFKA-2937
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Mayuresh Gharat
>
> In our clusters, we occasionally see topics marked for delete, but never 
> actually deleted. It may be due to brokers being restarted while tests were 
> running, but further restarts of Kafka dont fix the problem. The topics 
> remain marked for delete in Zookeeper.
> Topic describe shows:
> {quote}
> Topic:testtopic   PartitionCount:1ReplicationFactor:3 Configs:
>   Topic: testtopicPartition: 0Leader: noneReplicas: 3,4,0 
> Isr: 
> {quote}
> Kafka logs show:
> {quote}
> 2015-12-02 15:53:30,152] ERROR Controller 2 epoch 213 initiated state change 
> of replica 3 for partition [testtopic,0] from OnlineReplica to OfflineReplica 
> failed (state.change.logger)
> kafka.common.StateChangeFailedException: Failed to change state of replica 3 
> for partition [testtopic,0] since the leader and isr path in zookeeper is 
> empty
> at 
> kafka.controller.ReplicaStateMachine.handleStateChange(ReplicaStateMachine.scala:269)
> at 
> kafka.controller.ReplicaStateMachine$$anonfun$handleStateChanges$2.apply(ReplicaStateMachine.scala:114)
> at 
> kafka.controller.ReplicaStateMachine$$anonfun$handleStateChanges$2.apply(ReplicaStateMachine.scala:114)
> at 
> scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:322)
> at 
> scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:978)
> at 
> kafka.controller.ReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:114)
> at 
> kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2.apply(TopicDeletionManager.scala:342)
> at 
> kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2.apply(TopicDeletionManager.scala:334)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
> at 
> kafka.controller.TopicDeletionManager.startReplicaDeletion(TopicDeletionManager.scala:334)
> at 
> kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$onPartitionDeletion(TopicDeletionManager.scala:367)
> at 
> kafka.controller.TopicDeletionManager$$anonfun$kafka$controller$TopicDeletionManager$$onTopicDeletion$2.apply(TopicDeletionManager.scala:313)
> at 
> kafka.controller.TopicDeletionManager$$anonfun$kafka$controller$TopicDeletionManager$$onTopicDeletion$2.apply(TopicDeletionManager.scala:312)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
> at 
> kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$onTopicDeletion(TopicDeletionManager.scala:312)
> at 
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1$$anonfun$apply$mcV$sp$4.apply(TopicDeletionManager.scala:431)
> at 
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1$$anonfun$apply$mcV$sp$4.apply(TopicDeletionManager.scala:403)
> at scala.collection.immutable.Set$Set2.foreach(Set.scala:111)
> at 
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply$mcV$sp(TopicDeletionManager.scala:403)
> at 
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply(TopicDeletionManager.scala:397)
> at 
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply(TopicDeletionManager.scala:397)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
> at 
> kafka.controller.TopicDeletionManager$DeleteTopicsThread.doWork(TopicDeletionManager.scala:397)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> {quote}  
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2939) Make AbstractConfig.logUnused() tunable for clients

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-2939:


 Summary: Make AbstractConfig.logUnused() tunable for clients
 Key: KAFKA-2939
 URL: https://issues.apache.org/jira/browse/KAFKA-2939
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang
Assignee: Guozhang Wang
 Fix For: 0.9.1.0


Today we always log unused configs in KafkaProducer / KafkaConsumer in their 
constructors, however for some cases like Kafka Streams that make use of these 
clients, other configs may be passed in to configure Partitioner / Serializer 
classes, etc. So it would be better to make this function call optional to 
avoid printing unnecessary and confusing WARN entries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2914) Kafka Connect Source connector for HBase

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035439#comment-15035439
 ] 

Ewen Cheslack-Postava commented on KAFKA-2914:
--

[~nielsbasjes] Agreed that an HBase source connector would be great, and thanks 
for the pointer on how other projects grab the WAL. I think something like this 
is probably the right way to hook into HBase since it gives you the complete 
picture and probably gives the most flexibility wrt how to translate the WAL 
into messages in Kafka.

The plan was to keep the connector development federated, which means 
connectors like this would generally be maintained outside Kafka's source tree. 
This is partly just a practical decision, since pulling in a large variety of 
connectors would drastically complicate Kafka, its packaging, and its release 
process. But it also has nice side effects like decoupling connector release 
schedules from Kafka's, such that connectors can iterate more quickly than 
Kafka itself.

We have one very simple set of connectors implemented in Kafka for 
demonstration purposes, and while we do have KAFKA-2375 filed for an 
elasticsearch connector, we really only used it as a possible example to 
include in Kafka itself since it would be a more realistic example that doesn't 
have any extra dependencies.

I think adding an HBase connector would be hugely valuable, but should probably 
be done outside Kafka. I'll circle back soon with a template repository that 
can be used to bootstrap new connectors. This would be a good starting point 
for an HBase connector.

> Kafka Connect Source connector for HBase 
> -
>
> Key: KAFKA-2914
> URL: https://issues.apache.org/jira/browse/KAFKA-2914
> Project: Kafka
>  Issue Type: New Feature
>  Components: copycat
>Reporter: Niels Basjes
>Assignee: Ewen Cheslack-Postava
>
> In many cases I see HBase being used to persist data.
> I would like to listen to the changes and process them in a streaming system 
> (like Apache Flink).
> Feature request: A Kafka Connect "Source" that listens to the changes in a 
> specified HBase table. These changes are then stored in a 'standardized' form 
> in Kafka so that it becomes possible to process the observed changes in 
> near-realtime. I expect this 'standard' to be very HBase specific.
> Implementation suggestion: Perhaps listening to the HBase WAL like the "HBase 
> Side Effects Processor" does?
> https://github.com/NGDATA/hbase-indexer/tree/master/hbase-sep



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2908) Another, possibly different, Gap in Consumption after Restart

[Ben Stopford (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Stopford resolved KAFKA-2908.
-
Resolution: Fixed

Was bug in the way the test defines min.insync.replicas as found by [~rsivaram]

> Another, possibly different, Gap in Consumption after Restart
> -
>
> Key: KAFKA-2908
> URL: https://issues.apache.org/jira/browse/KAFKA-2908
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ben Stopford
>Assignee: Jason Gustafson
> Attachments: 2015-11-28--001.tar.gz
>
>
> *Context:*
> Instance of the rolling upgrade test. 10s sleeps have been put around node 
> restarts to ensure stability when subsequent nodes go down. Consumer timeout 
> has been set to 60s. Proudcer has been throttled to 100 messages per second. 
> Failure is rare: It occurred once in 60 executions (6 executions per run #276 
> -> #285 of the system_test_branch_builder)
> *Reported Failure:*
> At least one acked message did not appear in the consumed messages. 
> acked_minus_consumed: 16385, 16388, 16391, 16394, 16397, 16400, 16403, 16406, 
> 16409, 16412, 16415, 16418, 16421, 16424, 16427, 16430, 16433, 16436, 16439, 
> 16442, ...plus 1669 more
> *Immediate Observations:*
> * The list of messages not consumed are all in partition 1.
> * Production and Consumption continues throughout the test (there is no 
> complete write or read failure as we have seen elsewhere)
> * The messages ARE present in the data files:
> e.g. 
> {quote}
> Examining missing value 16385. This was written to p1,offset:5453
> => there is an entry in all data files for partition 1 for this (presumably 
> meaning it was replicated correctly)
> kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 
> worker10/kafka-data-logs/test_topic-1/.log | grep 
> 'offset: 5453'
> worker10,9,8:
> offset: 5453 position: 165346 isvalid: true payloadsize: 5 magic: 0 
> compresscodec: NoCompressionCodec crc: 1502953075 payload: 16385
> => ID 16385 is definitely present in the Kafka logs suggesting a problem 
> consumer-side
> {quote}
> These entries definitely do not appear in the consumer stdout file either. 
> *Timeline:*
> {quote}
> 15:27:02,232 - Producer sends first message
> ..servers 1, 2 are restarted (clean shutdown)
> 15:29:42,718 - Server 3 shutdown complete
> 15:29:42,712 - (Controller fails over): Broker 2 starting become controller 
> state transition (kafka.controller.KafkaController)
> 15:29:42,743 - New leasder is 2 (LeaderChangeListner)
> 15:29:43,239 - WARN Broker 2 ignoring LeaderAndIsr request from controller 2 
> with correlation id 0 epoch 7 for partition (test_topic,1) since its 
> associated leader epoch 8 is old. Current leader epoch is 8 
> (state.change.logger)
> 15:29:45,642 - Producer starts writing messages that are never consumed
> 15:30:10,804 - Last message sent by producer
> 15:31:10,983 - Consumer times out after 60s wait without messages
> {quote}
> Logs for this run are attached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (KAFKA-2908) Another, possibly different, Gap in Consumption after Restart

[Ben Stopford (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Stopford updated KAFKA-2908:

Comment: was deleted

(was: Was bug in the way the test defines min.insync.replicas as found by 
[~rsivaram])

> Another, possibly different, Gap in Consumption after Restart
> -
>
> Key: KAFKA-2908
> URL: https://issues.apache.org/jira/browse/KAFKA-2908
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ben Stopford
>Assignee: Jason Gustafson
> Attachments: 2015-11-28--001.tar.gz
>
>
> *Context:*
> Instance of the rolling upgrade test. 10s sleeps have been put around node 
> restarts to ensure stability when subsequent nodes go down. Consumer timeout 
> has been set to 60s. Proudcer has been throttled to 100 messages per second. 
> Failure is rare: It occurred once in 60 executions (6 executions per run #276 
> -> #285 of the system_test_branch_builder)
> *Reported Failure:*
> At least one acked message did not appear in the consumed messages. 
> acked_minus_consumed: 16385, 16388, 16391, 16394, 16397, 16400, 16403, 16406, 
> 16409, 16412, 16415, 16418, 16421, 16424, 16427, 16430, 16433, 16436, 16439, 
> 16442, ...plus 1669 more
> *Immediate Observations:*
> * The list of messages not consumed are all in partition 1.
> * Production and Consumption continues throughout the test (there is no 
> complete write or read failure as we have seen elsewhere)
> * The messages ARE present in the data files:
> e.g. 
> {quote}
> Examining missing value 16385. This was written to p1,offset:5453
> => there is an entry in all data files for partition 1 for this (presumably 
> meaning it was replicated correctly)
> kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 
> worker10/kafka-data-logs/test_topic-1/.log | grep 
> 'offset: 5453'
> worker10,9,8:
> offset: 5453 position: 165346 isvalid: true payloadsize: 5 magic: 0 
> compresscodec: NoCompressionCodec crc: 1502953075 payload: 16385
> => ID 16385 is definitely present in the Kafka logs suggesting a problem 
> consumer-side
> {quote}
> These entries definitely do not appear in the consumer stdout file either. 
> *Timeline:*
> {quote}
> 15:27:02,232 - Producer sends first message
> ..servers 1, 2 are restarted (clean shutdown)
> 15:29:42,718 - Server 3 shutdown complete
> 15:29:42,712 - (Controller fails over): Broker 2 starting become controller 
> state transition (kafka.controller.KafkaController)
> 15:29:42,743 - New leasder is 2 (LeaderChangeListner)
> 15:29:43,239 - WARN Broker 2 ignoring LeaderAndIsr request from controller 2 
> with correlation id 0 epoch 7 for partition (test_topic,1) since its 
> associated leader epoch 8 is old. Current leader epoch is 8 
> (state.change.logger)
> 15:29:45,642 - Producer starts writing messages that are never consumed
> 15:30:10,804 - Last message sent by producer
> 15:31:10,983 - Consumer times out after 60s wait without messages
> {quote}
> Logs for this run are attached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1894) Avoid long or infinite blocking in the consumer

[The Data Lorax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035665#comment-15035665
 ] 

The Data Lorax commented on KAFKA-1894:
---

I'm running into this issue and struggling to find a way around it - if the 
Kafka cluster is unavailable the KafkaConsumer.poll() call can block 
indefinitely - and does not even enter an interruptible state, which means 
there is no way of recovering, short of thread.stop().

Would be good to move this into a more imminent release or at least have the 
thread enter an interruptible state within the loop.

> Avoid long or infinite blocking in the consumer
> ---
>
> Key: KAFKA-1894
> URL: https://issues.apache.org/jira/browse/KAFKA-1894
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>Assignee: Jason Gustafson
> Fix For: 0.10.0.0
>
>
> The new consumer has a lot of loops that look something like
> {code}
>   while(!isThingComplete())
> client.poll();
> {code}
> This occurs both in KafkaConsumer but also in NetworkClient.completeAll. 
> These retry loops are actually mostly the behavior we want but there are 
> several cases where they may cause problems:
>  - In the case of a hard failure we may hang for a long time or indefinitely 
> before realizing the connection is lost.
>  - In the case where the cluster is malfunctioning or down we may retry 
> forever.
> It would probably be better to give a timeout to these. The proposed approach 
> would be to add something like retry.time.ms=6 and only continue retrying 
> for that period of time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2908) Consumer Sporadically Stops Consuming From Partition After Server Restart

[Ben Stopford (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Stopford updated KAFKA-2908:

Summary: Consumer Sporadically Stops Consuming From Partition After Server 
Restart  (was: Another, possibly different, Gap in Consumption after Restart)

> Consumer Sporadically Stops Consuming From Partition After Server Restart
> -
>
> Key: KAFKA-2908
> URL: https://issues.apache.org/jira/browse/KAFKA-2908
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ben Stopford
>Assignee: Jason Gustafson
> Attachments: 2015-11-28--001.tar.gz
>
>
> *Context:*
> Instance of the rolling upgrade test. 10s sleeps have been put around node 
> restarts to ensure stability when subsequent nodes go down. Consumer timeout 
> has been set to 60s. Proudcer has been throttled to 100 messages per second. 
> Failure is rare: It occurred once in 60 executions (6 executions per run #276 
> -> #285 of the system_test_branch_builder)
> *Reported Failure:*
> At least one acked message did not appear in the consumed messages. 
> acked_minus_consumed: 16385, 16388, 16391, 16394, 16397, 16400, 16403, 16406, 
> 16409, 16412, 16415, 16418, 16421, 16424, 16427, 16430, 16433, 16436, 16439, 
> 16442, ...plus 1669 more
> *Immediate Observations:*
> * The list of messages not consumed are all in partition 1.
> * Production and Consumption continues throughout the test (there is no 
> complete write or read failure as we have seen elsewhere)
> * The messages ARE present in the data files:
> e.g. 
> {quote}
> Examining missing value 16385. This was written to p1,offset:5453
> => there is an entry in all data files for partition 1 for this (presumably 
> meaning it was replicated correctly)
> kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 
> worker10/kafka-data-logs/test_topic-1/.log | grep 
> 'offset: 5453'
> worker10,9,8:
> offset: 5453 position: 165346 isvalid: true payloadsize: 5 magic: 0 
> compresscodec: NoCompressionCodec crc: 1502953075 payload: 16385
> => ID 16385 is definitely present in the Kafka logs suggesting a problem 
> consumer-side
> {quote}
> These entries definitely do not appear in the consumer stdout file either. 
> *Timeline:*
> {quote}
> 15:27:02,232 - Producer sends first message
> ..servers 1, 2 are restarted (clean shutdown)
> 15:29:42,718 - Server 3 shutdown complete
> 15:29:42,712 - (Controller fails over): Broker 2 starting become controller 
> state transition (kafka.controller.KafkaController)
> 15:29:42,743 - New leasder is 2 (LeaderChangeListner)
> 15:29:43,239 - WARN Broker 2 ignoring LeaderAndIsr request from controller 2 
> with correlation id 0 epoch 7 for partition (test_topic,1) since its 
> associated leader epoch 8 is old. Current leader epoch is 8 
> (state.change.logger)
> 15:29:45,642 - Producer starts writing messages that are never consumed
> 15:30:10,804 - Last message sent by producer
> 15:31:10,983 - Consumer times out after 60s wait without messages
> {quote}
> Logs for this run are attached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2909) Example of Gap in Consumption after Restart

[Ben Stopford (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035688#comment-15035688
 ] 

Ben Stopford commented on KAFKA-2909:
-

Was bug in the way the test defines min.insync.replicas as found by [~rsivaram]

> Example of Gap in Consumption after Restart
> ---
>
> Key: KAFKA-2909
> URL: https://issues.apache.org/jira/browse/KAFKA-2909
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ben Stopford
>Assignee: Jason Gustafson
>
> This seems very similar to Rajini's reported KAFAK-2891
> *Context*
> The context is Seurity Rolling Upgrade with 30s consumer timeout. There was a 
> 2s sleep between restarts. Throughput was limited to 1000 messages per 
> second. 
> *Failure*
> At least one acked message did not appear in the consumed messages. 
> acked_minus_consumed: set(36802, 36804, 36805, 36807, 36808, 36810, 36811, 
> 64403, 64406, 64409, 36799)
> Missing data was correctly written to Kafka data files:
> {quote}
> value 36802 -> partition 1,offset: 12216
> kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 
> worker7/kafka-data-logs/test_topic-1/.log | grep 'offset: 
> 12216'
> ->offset: 12216 position: 374994 isvalid: true payloadsize: 5 magic: 0 
> compresscodec: NoCompressionCodec crc: 3001177408 payload: 47482
> In this case offset 12216 does not contain the value we expected. The 
> implication is that the data was either not written or was lost during 
> rebalancing. 
> {quote}
> The first missing value was written at: 20:42:30,185, which is around the 
> time the third node goes down. 
> The failed writes correlate with the consumer logging out 
> NOT_COORDINATOR_FOR_GROUP and Marking the coordinator. There are many of 
> these messages though over a long period so it’s hard to infer this as being 
> the cause or specifically correlating with the error. 
> *Timeline*
> {quote}
> grep -r 'shutdown complete' *
> 20:42:06,132 - Node 1 shutdown completed 
> 20:42:18,560 - Node 2 shutdown completed 
> 20:42:30,185 - *Writes that never make it are written by producer*
> 20:42:31,164 - Node 3 shutdown completed 
> 20:42:57,872 - Node 1 shutdown completed 
> …
> {quote}
> Logging was verbose during this run so log files can be found here: [click 
> me|https://www.dropbox.com/s/owwzh37cs304qh4/2015-11-26--001%20%283%29.tar.gz?dl=0]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (KAFKA-2908) Another, possibly different, Gap in Consumption after Restart

[Ben Stopford (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Stopford reopened KAFKA-2908:
-

> Another, possibly different, Gap in Consumption after Restart
> -
>
> Key: KAFKA-2908
> URL: https://issues.apache.org/jira/browse/KAFKA-2908
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ben Stopford
>Assignee: Jason Gustafson
> Attachments: 2015-11-28--001.tar.gz
>
>
> *Context:*
> Instance of the rolling upgrade test. 10s sleeps have been put around node 
> restarts to ensure stability when subsequent nodes go down. Consumer timeout 
> has been set to 60s. Proudcer has been throttled to 100 messages per second. 
> Failure is rare: It occurred once in 60 executions (6 executions per run #276 
> -> #285 of the system_test_branch_builder)
> *Reported Failure:*
> At least one acked message did not appear in the consumed messages. 
> acked_minus_consumed: 16385, 16388, 16391, 16394, 16397, 16400, 16403, 16406, 
> 16409, 16412, 16415, 16418, 16421, 16424, 16427, 16430, 16433, 16436, 16439, 
> 16442, ...plus 1669 more
> *Immediate Observations:*
> * The list of messages not consumed are all in partition 1.
> * Production and Consumption continues throughout the test (there is no 
> complete write or read failure as we have seen elsewhere)
> * The messages ARE present in the data files:
> e.g. 
> {quote}
> Examining missing value 16385. This was written to p1,offset:5453
> => there is an entry in all data files for partition 1 for this (presumably 
> meaning it was replicated correctly)
> kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 
> worker10/kafka-data-logs/test_topic-1/.log | grep 
> 'offset: 5453'
> worker10,9,8:
> offset: 5453 position: 165346 isvalid: true payloadsize: 5 magic: 0 
> compresscodec: NoCompressionCodec crc: 1502953075 payload: 16385
> => ID 16385 is definitely present in the Kafka logs suggesting a problem 
> consumer-side
> {quote}
> These entries definitely do not appear in the consumer stdout file either. 
> *Timeline:*
> {quote}
> 15:27:02,232 - Producer sends first message
> ..servers 1, 2 are restarted (clean shutdown)
> 15:29:42,718 - Server 3 shutdown complete
> 15:29:42,712 - (Controller fails over): Broker 2 starting become controller 
> state transition (kafka.controller.KafkaController)
> 15:29:42,743 - New leasder is 2 (LeaderChangeListner)
> 15:29:43,239 - WARN Broker 2 ignoring LeaderAndIsr request from controller 2 
> with correlation id 0 epoch 7 for partition (test_topic,1) since its 
> associated leader epoch 8 is old. Current leader epoch is 8 
> (state.change.logger)
> 15:29:45,642 - Producer starts writing messages that are never consumed
> 15:30:10,804 - Last message sent by producer
> 15:31:10,983 - Consumer times out after 60s wait without messages
> {quote}
> Logs for this run are attached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2909) Example of Gap in Consumption after Restart

[Ben Stopford (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Stopford resolved KAFKA-2909.
-
Resolution: Fixed

> Example of Gap in Consumption after Restart
> ---
>
> Key: KAFKA-2909
> URL: https://issues.apache.org/jira/browse/KAFKA-2909
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ben Stopford
>Assignee: Jason Gustafson
>
> This seems very similar to Rajini's reported KAFAK-2891
> *Context*
> The context is Seurity Rolling Upgrade with 30s consumer timeout. There was a 
> 2s sleep between restarts. Throughput was limited to 1000 messages per 
> second. 
> *Failure*
> At least one acked message did not appear in the consumed messages. 
> acked_minus_consumed: set(36802, 36804, 36805, 36807, 36808, 36810, 36811, 
> 64403, 64406, 64409, 36799)
> Missing data was correctly written to Kafka data files:
> {quote}
> value 36802 -> partition 1,offset: 12216
> kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 
> worker7/kafka-data-logs/test_topic-1/.log | grep 'offset: 
> 12216'
> ->offset: 12216 position: 374994 isvalid: true payloadsize: 5 magic: 0 
> compresscodec: NoCompressionCodec crc: 3001177408 payload: 47482
> In this case offset 12216 does not contain the value we expected. The 
> implication is that the data was either not written or was lost during 
> rebalancing. 
> {quote}
> The first missing value was written at: 20:42:30,185, which is around the 
> time the third node goes down. 
> The failed writes correlate with the consumer logging out 
> NOT_COORDINATOR_FOR_GROUP and Marking the coordinator. There are many of 
> these messages though over a long period so it’s hard to infer this as being 
> the cause or specifically correlating with the error. 
> *Timeline*
> {quote}
> grep -r 'shutdown complete' *
> 20:42:06,132 - Node 1 shutdown completed 
> 20:42:18,560 - Node 2 shutdown completed 
> 20:42:30,185 - *Writes that never make it are written by producer*
> 20:42:31,164 - Node 3 shutdown completed 
> 20:42:57,872 - Node 1 shutdown completed 
> …
> {quote}
> Logging was verbose during this run so log files can be found here: [click 
> me|https://www.dropbox.com/s/owwzh37cs304qh4/2015-11-26--001%20%283%29.tar.gz?dl=0]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2935) Remove vestigial CLUSTER_CONFIG in WorkerConfig

[Ewen Cheslack-Postava (JIRA)]

Ewen Cheslack-Postava created KAFKA-2935:


 Summary: Remove vestigial CLUSTER_CONFIG in WorkerConfig 
 Key: KAFKA-2935
 URL: https://issues.apache.org/jira/browse/KAFKA-2935
 Project: Kafka
  Issue Type: Bug
  Components: copycat
Affects Versions: 0.9.0.0
Reporter: Ewen Cheslack-Postava
Assignee: Ewen Cheslack-Postava


This config isn't used anywhere anymore. Its previous reason for existence is 
now handled by DistributedConfig.GROUP_ID_CONFIG.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2642) Run replication tests in ducktape with SSL for clients

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035835#comment-15035835
 ] 

ASF GitHub Bot commented on KAFKA-2642:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/563


> Run replication tests in ducktape with SSL for clients
> --
>
> Key: KAFKA-2642
> URL: https://issues.apache.org/jira/browse/KAFKA-2642
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.1.0
>
>
> Under KAFKA-2581, replication tests were parametrized to run with SSL for 
> interbroker communication, but not for clients. When KAFKA-2603 is committed, 
> the tests should be able to use SSL for clients as well,



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2642) Run replication tests in ducktape with SSL for clients

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-2642.

Resolution: Fixed

Issue resolved by pull request 563
[https://github.com/apache/kafka/pull/563]

> Run replication tests in ducktape with SSL for clients
> --
>
> Key: KAFKA-2642
> URL: https://issues.apache.org/jira/browse/KAFKA-2642
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.1.0
>
>
> Under KAFKA-2581, replication tests were parametrized to run with SSL for 
> interbroker communication, but not for clients. When KAFKA-2603 is committed, 
> the tests should be able to use SSL for clients as well,



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2642: Run replication tests with SSL and...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/563


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-2940) Make available to use any Java options at startup scripts

[Sasaki Toru (JIRA)]

Sasaki Toru created KAFKA-2940:
--

 Summary: Make available to use any Java options at startup scripts
 Key: KAFKA-2940
 URL: https://issues.apache.org/jira/browse/KAFKA-2940
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.9.0.0
Reporter: Sasaki Toru
Priority: Minor
 Fix For: 0.9.0.1


We cannot specify any Java options (e.g. option for remote debugging) at 
startup scrips such as kafka-server-start.sh .
This ticket makes we can specify to use "JAVA_OPTS" environmental variables.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)