[jira] [Commented] (KAFKA-1344) Kafka-console-producer.sh support snappy compression
[ https://issues.apache.org/jira/browse/KAFKA-1344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955419#comment-13955419 ] ASF GitHub Bot commented on KAFKA-1344: --- GitHub user edgefox opened a pull request: https://github.com/apache/kafka/pull/20 [KAFKA-1344] - Added compression codec option feature for ConsoleProducer [KAFKA-1344] - Added compression codec option feature for ConsoleProducer You can merge this pull request into a Git repository by running: $ git pull https://github.com/edgefox/kafka 0.8.1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/20.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20 commit ca241fd6d1b918a9ea5f49c6a8d35abe8f420a0b Author: edgefox ivanlyu...@gmail.com Date: 2014-03-31T16:37:46Z [KAFKA-1344] - Added compression codec option feature for ConsoleProducer Kafka-console-producer.sh support snappy compression Key: KAFKA-1344 URL: https://issues.apache.org/jira/browse/KAFKA-1344 Project: Kafka Issue Type: Improvement Reporter: aio Labels: tools, usability Fix For: 0.8.2 Wish kafka-console-producer.sh support snappy compression. Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1370) Gradle startup script for Windows
[ https://issues.apache.org/jira/browse/KAFKA-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962800#comment-13962800 ] ASF GitHub Bot commented on KAFKA-1370: --- GitHub user sslavic opened a pull request: https://github.com/apache/kafka/pull/22 KAFKA-1370 Added Gradle startup script for Windows This patch adds Gradle startup script for Windows You can merge this pull request into a Git repository by running: $ git pull https://github.com/sslavic/kafka KAFKA-1370 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/22.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22 commit 2d47e088d7f47fd06e7842b444b9039715616358 Author: Stevo Slavic | hybris stevo.sla...@hybris.com Date: 2014-04-08T12:02:16Z KAFKA-1370 Added Gradle startup script for Windows Gradle startup script for Windows - Key: KAFKA-1370 URL: https://issues.apache.org/jira/browse/KAFKA-1370 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.8.1 Reporter: Stevo Slavic Priority: Trivial Labels: gradle Please provide Gradle startup script for Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1372) Upgrade to Gradle 1.11
[ https://issues.apache.org/jira/browse/KAFKA-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962809#comment-13962809 ] ASF GitHub Bot commented on KAFKA-1372: --- GitHub user sslavic opened a pull request: https://github.com/apache/kafka/pull/23 KAFKA-1372 Upgrade to Gradle 1.11 This patch upgrades Gradle wrapper from 1.6 to 1.11 You can merge this pull request into a Git repository by running: $ git pull https://github.com/sslavic/kafka KAFKA-1372 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/23.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23 commit 9f23afe3ef194dd61bb7ba36b28df98f3490e10b Author: Stevo Slavic | hybris stevo.sla...@hybris.com Date: 2014-04-08T12:19:45Z KAFKA-1372 Upgrade to Gradle 1.11 Upgrade to Gradle 1.11 -- Key: KAFKA-1372 URL: https://issues.apache.org/jira/browse/KAFKA-1372 Project: Kafka Issue Type: Task Components: tools Affects Versions: 0.8.1 Reporter: Stevo Slavic Priority: Minor Labels: gradle Currently used version of Gradle wrapper is 1.6 while 1.11 is available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1375) Formatting for Running a task on a particular version of Scala paragraph in README.md is broken
[ https://issues.apache.org/jira/browse/KAFKA-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964158#comment-13964158 ] ASF GitHub Bot commented on KAFKA-1375: --- GitHub user sslavic opened a pull request: https://github.com/apache/kafka/pull/24 KAFKA-1375: Fix formatting in README.md This patch fixes formatting of instructions for using particular version of Scala when running a Gradle build task. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sslavic/kafka KAFKA-1375 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/24.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #24 commit f3cfc76185d17b8a0970bb19bbb0d4979beec0dc Author: Stevo Slavic ssla...@gmail.com Date: 2014-04-09T13:50:13Z KAFKA-1375: Fixed formatting of instructions for using particular version of Scala when running a Gradle build task Formatting for Running a task on a particular version of Scala paragraph in README.md is broken - Key: KAFKA-1375 URL: https://issues.apache.org/jira/browse/KAFKA-1375 Project: Kafka Issue Type: Bug Components: website Affects Versions: 0.8.1 Reporter: Stevo Slavic Priority: Trivial Labels: documentation See commit which broke formatting at https://github.com/apache/kafka/commit/879e3e770ebc49f916137e8416df74373fa26a74#diff-04c6e90faac2675aa89e2176d2eec7d8 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1414) Speedup broker startup after hard reset
[
https://issues.apache.org/jira/browse/KAFKA-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062680#comment-14062680
]
ASF GitHub Bot commented on KAFKA-1414:
---
GitHub user ataraxer opened a pull request:
https://github.com/apache/kafka/pull/26
KAFKA-1414: Speedup broker startup after hard reset and shutdown
This patch increases speed of both hard reset and shutdown by introducing
`log.recovery.threads` and `log.shutdown.threads` properties, which allows to
perform work required for them in parallel, grained by log directories.
Best performance can be achieved by setting thread count to number of log
directories, provided that they are located on dedicated drives. Although that
option should be used with caution due to the possibility of native JVM out of
memory error.
Patch is compiled of changes proposed by Jay Kreps, Alexey Ozeritskiy,
Dmitry Bugaychenko by Anton Karamanov.
All tests are passing.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ataraxer/kafka kafka-1414
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/26.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #26
commit e4a86709d07030c44f077ab20d4329ddb84c4aec
Author: Anton Karamanov atara...@gmail.com
Date: 2014-07-15T17:42:15Z
KAFKA-1414 Speedup broker startup after hard reset and shutdown; patched by
Jay Kreps, Alexey Ozeritskiy, Dmitry Bugaychenko and Anton Karamanov
Speedup broker startup after hard reset
---
Key: KAFKA-1414
URL: https://issues.apache.org/jira/browse/KAFKA-1414
Project: Kafka
Issue Type: Improvement
Components: log
Affects Versions: 0.8.2, 0.9.0, 0.8.1.1
Reporter: Dmitry Bugaychenko
Assignee: Jay Kreps
Attachments: parallel-dir-loading-0.8.patch,
parallel-dir-loading-trunk-fixed-threadpool.patch,
parallel-dir-loading-trunk-threadpool.patch, parallel-dir-loading-trunk.patch
After hard reset due to power failure broker takes way too much time
recovering unflushed segments in a single thread. This could be easiliy
improved launching multiple threads (one per data dirrectory, assuming that
typically each data directory is on a dedicated drive). Localy we trie this
simple patch to LogManager.loadLogs and it seems to work, however I'm too new
to scala, so do not take it literally:
{code}
/**
* Recover and load all logs in the given data directories
*/
private def loadLogs(dirs: Seq[File]) {
val threads : Array[Thread] = new Array[Thread](dirs.size)
var i: Int = 0
val me = this
for(dir - dirs) {
val thread = new Thread( new Runnable {
def run()
{
val recoveryPoints = me.recoveryPointCheckpoints(dir).read
/* load the logs */
val subDirs = dir.listFiles()
if(subDirs != null) {
val cleanShutDownFile = new File(dir, Log.CleanShutdownFile)
if(cleanShutDownFile.exists())
info(Found clean shutdown file. Skipping recovery for all logs
in data directory '%s'.format(dir.getAbsolutePath))
for(dir - subDirs) {
if(dir.isDirectory) {
info(Loading log ' + dir.getName + ')
val topicPartition = Log.parseTopicPartitionName(dir.getName)
val config = topicConfigs.getOrElse(topicPartition.topic,
defaultConfig)
val log = new Log(dir,
config,
recoveryPoints.getOrElse(topicPartition, 0L),
scheduler,
time)
val previous = addLogWithLock(topicPartition, log)
if(previous != null)
throw new IllegalArgumentException(Duplicate log
directories found: %s, %s!.format(log.dir.getAbsolutePath,
previous.dir.getAbsolutePath))
}
}
cleanShutDownFile.delete()
}
}
})
thread.start()
threads(i) = thread
i = i + 1
}
for(thread - threads) {
thread.join()
}
}
def addLogWithLock(topicPartition: TopicAndPartition, log: Log): Log = {
logCreationOrDeletionLock synchronized {
this.logs.put(topicPartition, log)
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (KAFKA-1559) Upgrade Gradle wrapper to Gradle 2.0
[ https://issues.apache.org/jira/browse/KAFKA-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14075603#comment-14075603 ] ASF GitHub Bot commented on KAFKA-1559: --- GitHub user sslavic opened a pull request: https://github.com/apache/kafka/pull/29 Upgrade Gradle wrapper to Gradle 2.0 This patch upgrades gradle wrapper to gradle 2.0. As consequence license plugin dependency had to be upgraded as well. Issue: KAFKA-1559 You can merge this pull request into a Git repository by running: $ git pull https://github.com/sslavic/kafka KAFKA-1559 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/29.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #29 commit c64fb1c7c61c4dd2a0ed0989f357caa9de1027a8 Author: Stevo Slavic ssla...@gmail.com Date: 2014-07-27T10:48:56Z KAFKA-1559: Upgraded gradle wrapper to gradle 2.0; as consequence license plugin dependency had to be upgraded as well Upgrade Gradle wrapper to Gradle 2.0 Key: KAFKA-1559 URL: https://issues.apache.org/jira/browse/KAFKA-1559 Project: Kafka Issue Type: Task Components: build Affects Versions: 0.8.1.1 Reporter: Stevo Slavic Priority: Trivial Labels: build -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1414) Speedup broker startup after hard reset
[
https://issues.apache.org/jira/browse/KAFKA-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14076660#comment-14076660
]
ASF GitHub Bot commented on KAFKA-1414:
---
Github user ataraxer closed the pull request at:
https://github.com/apache/kafka/pull/26
Speedup broker startup after hard reset
---
Key: KAFKA-1414
URL: https://issues.apache.org/jira/browse/KAFKA-1414
Project: Kafka
Issue Type: Improvement
Components: log
Affects Versions: 0.8.1.1, 0.8.2, 0.9.0
Reporter: Dmitry Bugaychenko
Assignee: Anton Karamanov
Fix For: 0.8.2
Attachments:
0001-KAFKA-1414-Speedup-broker-startup-after-hard-reset-a.patch,
KAFKA-1414-rev1.patch, KAFKA-1414-rev2.fixed.patch, KAFKA-1414-rev2.patch,
KAFKA-1414-rev3-interleaving.patch, KAFKA-1414-rev3.patch,
KAFKA-1414-rev4.patch, KAFKA-1414-rev5.patch, freebie.patch,
parallel-dir-loading-0.8.patch,
parallel-dir-loading-trunk-fixed-threadpool.patch,
parallel-dir-loading-trunk-threadpool.patch, parallel-dir-loading-trunk.patch
After hard reset due to power failure broker takes way too much time
recovering unflushed segments in a single thread. This could be easiliy
improved launching multiple threads (one per data dirrectory, assuming that
typically each data directory is on a dedicated drive). Localy we trie this
simple patch to LogManager.loadLogs and it seems to work, however I'm too new
to scala, so do not take it literally:
{code}
/**
* Recover and load all logs in the given data directories
*/
private def loadLogs(dirs: Seq[File]) {
val threads : Array[Thread] = new Array[Thread](dirs.size)
var i: Int = 0
val me = this
for(dir - dirs) {
val thread = new Thread( new Runnable {
def run()
{
val recoveryPoints = me.recoveryPointCheckpoints(dir).read
/* load the logs */
val subDirs = dir.listFiles()
if(subDirs != null) {
val cleanShutDownFile = new File(dir, Log.CleanShutdownFile)
if(cleanShutDownFile.exists())
info(Found clean shutdown file. Skipping recovery for all logs
in data directory '%s'.format(dir.getAbsolutePath))
for(dir - subDirs) {
if(dir.isDirectory) {
info(Loading log ' + dir.getName + ')
val topicPartition = Log.parseTopicPartitionName(dir.getName)
val config = topicConfigs.getOrElse(topicPartition.topic,
defaultConfig)
val log = new Log(dir,
config,
recoveryPoints.getOrElse(topicPartition, 0L),
scheduler,
time)
val previous = addLogWithLock(topicPartition, log)
if(previous != null)
throw new IllegalArgumentException(Duplicate log
directories found: %s, %s!.format(log.dir.getAbsolutePath,
previous.dir.getAbsolutePath))
}
}
cleanShutDownFile.delete()
}
}
})
thread.start()
threads(i) = thread
i = i + 1
}
for(thread - threads) {
thread.join()
}
}
def addLogWithLock(topicPartition: TopicAndPartition, log: Log): Log = {
logCreationOrDeletionLock synchronized {
this.logs.put(topicPartition, log)
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (KAFKA-1369) snappy version update 1.1.x
[
https://issues.apache.org/jira/browse/KAFKA-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102103#comment-14102103
]
ASF GitHub Bot commented on KAFKA-1369:
---
GitHub user thinker0 opened a pull request:
https://github.com/apache/kafka/pull/31
KAFKA-1369 - snappy version update 1.1.x
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/thinker0/kafka snappy-update
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/31.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #31
commit 214aa1a1cdcc89a73d9304d874d1074f63e78211
Author: thinker0 think...@daumcorp.com
Date: 2014-08-19T10:17:14Z
KAFKA-1369 - snappy version update 1.1.x
snappy version update 1.1.x
---
Key: KAFKA-1369
URL: https://issues.apache.org/jira/browse/KAFKA-1369
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.0
Environment: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
- x64
Reporter: thinker0
Priority: Minor
https://github.com/xerial/snappy-java/issues/38 issue
snappy version 1.1.x
{code}
org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:239)
at org.xerial.snappy.Snappy.clinit(Snappy.java:48)
at
org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:351)
at org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:159)
at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142)
at java.io.InputStream.read(InputStream.java:101)
at
kafka.message.ByteBufferMessageSet$$anonfun$decompress$1.apply$mcI$sp(ByteBufferMessageSet.scala:68)
at
kafka.message.ByteBufferMessageSet$$anonfun$decompress$1.apply(ByteBufferMessageSet.scala:68)
at
kafka.message.ByteBufferMessageSet$$anonfun$decompress$1.apply(ByteBufferMessageSet.scala:68)
at scala.collection.immutable.Stream$.continually(Stream.scala:1129)
at
kafka.message.ByteBufferMessageSet$.decompress(ByteBufferMessageSet.scala:68)
at
kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:178)
at
kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:191)
at
kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:145)
at
kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
{code}
{code}
/tmp] ldd snappy-1.0.5-libsnappyjava.so
./snappy-1.0.5-libsnappyjava.so: /usr/lib64/libstdc++.so.6: version
`GLIBCXX_3.4.9' not found (required by ./snappy-1.0.5-libsnappyjava.so)
./snappy-1.0.5-libsnappyjava.so: /usr/lib64/libstdc++.so.6: version
`GLIBCXX_3.4.11' not found (required by ./snappy-1.0.5-libsnappyjava.so)
linux-vdso.so.1 = (0x7fff81dfc000)
libstdc++.so.6 = /usr/lib64/libstdc++.so.6 (0x2b554b43)
libm.so.6 = /lib64/libm.so.6 (0x2b554b731000)
libc.so.6 = /lib64/libc.so.6 (0x2b554b9b4000)
libgcc_s.so.1 = /lib64/libgcc_s.so.1 (0x2b554bd0c000)
/lib64/ld-linux-x86-64.so.2 (0x0033e2a0)
{code}
{code}
/tmp] ldd snappy-1.1.1M1-be6ba593-9ac7-488e-953e-ba5fd9530ee1-libsnappyjava.so
ldd: warning: you do not have execution permission for
`./snappy-1.1.1M1-be6ba593-9ac7-488e-953e-ba5fd9530ee1-libsnappyjava.so'
linux-vdso.so.1 = (0x7fff1c132000)
libstdc++.so.6 = /usr/lib64/libstdc++.so.6 (0x2b9548319000)
libm.so.6 = /lib64/libm.so.6 (0x2b954861a000)
libc.so.6 = /lib64/libc.so.6 (0x2b954889d000)
libgcc_s.so.1 = /lib64/libgcc_s.so.1 (0x2b9548bf5000)
/lib64/ld-linux-x86-64.so.2 (0x0033e2a0)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (KAFKA-1635) Java doc of makeLeaders in ReplicaManager is wrong
[ https://issues.apache.org/jira/browse/KAFKA-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135135#comment-14135135 ] ASF GitHub Bot commented on KAFKA-1635: --- GitHub user LantaoJin opened a pull request: https://github.com/apache/kafka/pull/33 KAFKA-1635: Fixed incorrect java doc of makeLeaders() in ReplicaManager You can merge this pull request into a Git repository by running: $ git pull https://github.com/LantaoJin/kafka KAFKA-1635 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/33.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #33 commit 6739a8e601331ad07d9856dc351785351755a5d5 Author: LantaoJin lantao@dianping.com Date: 2014-09-16T08:26:34Z KAFKA-1635: Fixed incorrect java doc of makeLeaders() in ReplicaManager Java doc of makeLeaders in ReplicaManager is wrong -- Key: KAFKA-1635 URL: https://issues.apache.org/jira/browse/KAFKA-1635 Project: Kafka Issue Type: Bug Components: core, replication Reporter: Lantao Jin Assignee: Neha Narkhede Priority: Minor Labels: doc, server Attachments: kafka-1635-1.patch ReplicaManager have an incorrect java doc. The overview of function makeLeaders() is the same as makeFollowers(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1635) Java doc of makeLeaders in ReplicaManager is wrong
[ https://issues.apache.org/jira/browse/KAFKA-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14137044#comment-14137044 ] ASF GitHub Bot commented on KAFKA-1635: --- Github user LantaoJin closed the pull request at: https://github.com/apache/kafka/pull/33 Java doc of makeLeaders in ReplicaManager is wrong -- Key: KAFKA-1635 URL: https://issues.apache.org/jira/browse/KAFKA-1635 Project: Kafka Issue Type: Bug Components: core, replication Reporter: Lantao Jin Assignee: Lantao Jin Priority: Trivial Labels: doc, server Fix For: 0.8.2 Attachments: kafka-1635-1.patch ReplicaManager have an incorrect java doc. The overview of function makeLeaders() is the same as makeFollowers(). Also see commit at https://github.com/apache/kafka/commit/6739a8e601331ad07d9856dc351785351755a5d5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1833) OfflinePartitionLeaderSelector may return null leader when ISR and Assgined Broker have no common
[ https://issues.apache.org/jira/browse/KAFKA-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259911#comment-14259911 ] ASF GitHub Bot commented on KAFKA-1833: --- GitHub user tedxia opened a pull request: https://github.com/apache/kafka/pull/39 KAFKA-1833: OfflinePartitionLeaderSelector may return null leader when ISR and Assgi... In OfflinePartitonLeaderSelector::selectLeader, when liveBrokerInIsr is not empty and have no common broker with liveAssignedreplicas, selectLeader will return no leader; You can merge this pull request into a Git repository by running: $ git pull https://github.com/tedxia/kafka fix-select-leader Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/39.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #39 commit 6f20bdc9ebe7b4a56de492c98e7a89ac7e3985ba Author: xiajun xia...@xiaomi.com Date: 2014-12-29T07:22:54Z OfflinePartitionLeaderSelector may return null leader when ISR and Assgined Broker have no common broker OfflinePartitionLeaderSelector may return null leader when ISR and Assgined Broker have no common - Key: KAFKA-1833 URL: https://issues.apache.org/jira/browse/KAFKA-1833 Project: Kafka Issue Type: Bug Components: controller Affects Versions: 0.8.2 Reporter: xiajun Assignee: Neha Narkhede Labels: easyfix In OfflinePartitonLeaderSelector::selectLeader, when liveBrokerInIsr is not empty and have no common broker with liveAssignedreplicas, selectLeader will return no leader; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1834) No Response when handle LeaderAndIsrRequest some case
[ https://issues.apache.org/jira/browse/KAFKA-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259919#comment-14259919 ] ASF GitHub Bot commented on KAFKA-1834: --- GitHub user tedxia opened a pull request: https://github.com/apache/kafka/pull/40 KAFKA-1834: No Response when handle LeaderAndIsrRequest some case PR for [KAFKA-1834](https://issues.apache.org/jira/browse/KAFKA-1834) When a replica become leader or follower, if this broker no exist in assigned replicas, there are no response. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tedxia/kafka fix-noresponse-on-become-leader-or-follower Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/40.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #40 commit 80858f72c39eaa974a6085f78e797de4e8c55aae Author: xiajun xia...@xiaomi.com Date: 2014-12-29T07:53:44Z No Response when handle LeaderAndIsrRequest some case No Response when handle LeaderAndIsrRequest some case - Key: KAFKA-1834 URL: https://issues.apache.org/jira/browse/KAFKA-1834 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2 Reporter: xiajun Labels: easyfix When a replica become leader or follower, if this broker no exist in assigned replicas, there are no response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-873) Consider replacing zkclient with curator (with zkclient-bridge)
[ https://issues.apache.org/jira/browse/KAFKA-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387573#comment-14387573 ] ASF GitHub Bot commented on KAFKA-873: -- GitHub user atdixon opened a pull request: https://github.com/apache/kafka/pull/53 curator + exhibitor integration My motivation for introducing curator to kafka was to get optional exhibitor support, however I noticed this is also a solution to ticket KAFKA-873 (https://issues.apache.org/jira/browse/KAFKA-873). Structurally I believe the code is sound, however some tests are blocking which I believe is duet o races related to in-memory Zookeeper but not entirely sure. Am looking into it and testing outside of in-memory ZK, as well. But would love comments/discussion on this PR. I imagine exhibitor support is something that many are interested in, especially those of us in AWS cloud environments. You can merge this pull request into a Git repository by running: $ git pull https://github.com/atdixon/kafka exhibitor-support Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/53.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #53 commit 3526d5664fa87b2d3f0e6e35bd5b060639df4ead Author: Aaron Dixon atdi...@gmail.com Date: 2015-03-30T23:15:16Z curator + exhibitor integration Consider replacing zkclient with curator (with zkclient-bridge) --- Key: KAFKA-873 URL: https://issues.apache.org/jira/browse/KAFKA-873 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.0 Reporter: Scott Clasen Assignee: Grant Henke If zkclient was replaced with curator and curator-x-zkclient-bridge it would be initially a drop-in replacement https://github.com/Netflix/curator/wiki/ZKClient-Bridge With the addition of a few more props to ZkConfig, and a bit of code this would open up the possibility of using ACLs in zookeeper (which arent supported directly by zkclient), as well as integrating with netflix exhibitor for those of us using that. Looks like KafkaZookeeperClient needs some love anyhow... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1621) Standardize --messages option in perf scripts
[ https://issues.apache.org/jira/browse/KAFKA-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14332423#comment-14332423 ] ASF GitHub Bot commented on KAFKA-1621: --- GitHub user rekhajoshm opened a pull request: https://github.com/apache/kafka/pull/46 KAFKA-1621 : Standardize --messages option KAFKA-1621: Standardize --messages option in perf scripts You can merge this pull request into a Git repository by running: $ git pull https://github.com/rekhajoshm/kafka KAFKA-1621 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/46.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #46 commit d123f48f85604c765b464d6b9d5cee4b3ec0de25 Author: Joshi rekhajo...@gmail.com Date: 2015-02-22T23:21:55Z KAFKA-1621 : Standardize --messages option Standardize --messages option in perf scripts - Key: KAFKA-1621 URL: https://issues.apache.org/jira/browse/KAFKA-1621 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1.1 Reporter: Jay Kreps Labels: newbie This option is specified in PerfConfig and is used by the producer, consumer and simple consumer perf commands. The docstring on the argument does not list it as required but the producer performance test requires it--others don't. We should standardize this so that either all the commands require the option and it is marked as required in the docstring or none of them list it as required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1972) JMX Tool output for CSV format does not handle attributes with comma in their value
[ https://issues.apache.org/jira/browse/KAFKA-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14332386#comment-14332386 ] ASF GitHub Bot commented on KAFKA-1972: --- GitHub user rekhajoshm opened a pull request: https://github.com/apache/kafka/pull/45 KAFKA-1972: JMXTool multiple attributes KAFKA-1972: JMX Tool output for CSV format does not handle attributes with comma in their value You can merge this pull request into a Git repository by running: $ git pull https://github.com/rekhajoshm/kafka KAFKA-1972 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/45.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #45 commit b599610ef512c21f5acb621c65168da03c8093c0 Author: Joshi rekhajo...@gmail.com Date: 2015-02-22T21:48:55Z KAFKA-1972: JMXTool multiple attributes JMX Tool output for CSV format does not handle attributes with comma in their value --- Key: KAFKA-1972 URL: https://issues.apache.org/jira/browse/KAFKA-1972 Project: Kafka Issue Type: Bug Components: tools Affects Versions: 0.8.1.1 Reporter: Jonathan Rafalski Priority: Minor Labels: newbie When the JMXTools outputs all attributes using a comma delimitation it does not have an exit character or a way to handle attributes that contain comma's in their value. This could potentially limit the uses of the output to single value attributes only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1545) java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may fail on some irregular hostnames
[ https://issues.apache.org/jira/browse/KAFKA-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14332432#comment-14332432 ] ASF GitHub Bot commented on KAFKA-1545: --- GitHub user rekhajoshm opened a pull request: https://github.com/apache/kafka/pull/47 KAFKA-1545: KafkaHealthcheck.register failure KAFKA-1545: java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may fail on some irregular hostnames You can merge this pull request into a Git repository by running: $ git pull https://github.com/rekhajoshm/kafka KAFKA-1545 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/47.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #47 commit d123f48f85604c765b464d6b9d5cee4b3ec0de25 Author: Joshi rekhajo...@gmail.com Date: 2015-02-22T23:21:55Z KAFKA-1621 : Standardize --messages option commit 262df13b91d86bee2c5fb937630c794830854947 Author: Joshi rekhajo...@gmail.com Date: 2015-02-22T23:54:43Z KAFKA-1545: KafkaHealthcheck.register failure java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may fail on some irregular hostnames --- Key: KAFKA-1545 URL: https://issues.apache.org/jira/browse/KAFKA-1545 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Labels: newbie Fix For: 0.9.0 For example: kafka.server.LogOffsetTest testGetOffsetsForUnknownTopic FAILED java.net.UnknownHostException: guwang-mn2: guwang-mn2: nodename nor servname provided, or not known at java.net.InetAddress.getLocalHost(InetAddress.java:1473) at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:59) at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:45) at kafka.server.KafkaServer.startup(KafkaServer.scala:121) at kafka.utils.TestUtils$.createServer(TestUtils.scala:130) at kafka.server.LogOffsetTest.setUp(LogOffsetTest.scala:53) Caused by: java.net.UnknownHostException: guwang-mn2: nodename nor servname provided, or not known at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293) at java.net.InetAddress.getLocalHost(InetAddress.java:1469) ... 5 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1545) java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may fail on some irregular hostnames
[ https://issues.apache.org/jira/browse/KAFKA-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14332433#comment-14332433 ] ASF GitHub Bot commented on KAFKA-1545: --- Github user rekhajoshm closed the pull request at: https://github.com/apache/kafka/pull/47 java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may fail on some irregular hostnames --- Key: KAFKA-1545 URL: https://issues.apache.org/jira/browse/KAFKA-1545 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Labels: newbie Fix For: 0.9.0 For example: kafka.server.LogOffsetTest testGetOffsetsForUnknownTopic FAILED java.net.UnknownHostException: guwang-mn2: guwang-mn2: nodename nor servname provided, or not known at java.net.InetAddress.getLocalHost(InetAddress.java:1473) at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:59) at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:45) at kafka.server.KafkaServer.startup(KafkaServer.scala:121) at kafka.utils.TestUtils$.createServer(TestUtils.scala:130) at kafka.server.LogOffsetTest.setUp(LogOffsetTest.scala:53) Caused by: java.net.UnknownHostException: guwang-mn2: nodename nor servname provided, or not known at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293) at java.net.InetAddress.getLocalHost(InetAddress.java:1469) ... 5 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-269) ./system_test/producer_perf/bin/run-test.sh without --async flag does not run
[ https://issues.apache.org/jira/browse/KAFKA-269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14332475#comment-14332475 ] ASF GitHub Bot commented on KAFKA-269: -- GitHub user rekhajoshm opened a pull request: https://github.com/apache/kafka/pull/49 KAFKA-269: run-test.sh async test KAFKA-269: run-test.sh async test You can merge this pull request into a Git repository by running: $ git pull https://github.com/rekhajoshm/kafka KAFKA-269 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/49.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #49 commit 99302738459c1be9166ca9808971643bc220f675 Author: Joshi rekhajo...@gmail.com Date: 2015-02-23T01:38:29Z KAFKA-269: run-test.sh async test ./system_test/producer_perf/bin/run-test.sh without --async flag does not run -- Key: KAFKA-269 URL: https://issues.apache.org/jira/browse/KAFKA-269 Project: Kafka Issue Type: Bug Components: clients, core Affects Versions: 0.7 Environment: Linux 2.6.18-238.1.1.el5 , x86_64 x86_64 x86_64 GNU/Linux ext3 file system with raid10 Reporter: Praveen Ramachandra Labels: newbie, performance When I run the tests without --async option, The tests doesn't produce even a single message. Following defaults where changed in the server.properties num.threads=Tried with 8, 10, 100 num.partitions=10 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1545) java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may fail on some irregular hostnames
[ https://issues.apache.org/jira/browse/KAFKA-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14332435#comment-14332435 ] ASF GitHub Bot commented on KAFKA-1545: --- GitHub user rekhajoshm opened a pull request: https://github.com/apache/kafka/pull/48 KAFKA-1545: KafkaHealthcheck.register failure KAFKA-1545: java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may fail on some irregular hostnames You can merge this pull request into a Git repository by running: $ git pull https://github.com/rekhajoshm/kafka KAFKA-1545 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/48.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #48 commit 3127b9058b19916657c234635437edf8a93123d4 Author: Joshi rekhajo...@gmail.com Date: 2015-02-23T00:05:28Z KAFKA-1545 : KafkaHealthcheck.register failure java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may fail on some irregular hostnames --- Key: KAFKA-1545 URL: https://issues.apache.org/jira/browse/KAFKA-1545 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Labels: newbie Fix For: 0.9.0 For example: kafka.server.LogOffsetTest testGetOffsetsForUnknownTopic FAILED java.net.UnknownHostException: guwang-mn2: guwang-mn2: nodename nor servname provided, or not known at java.net.InetAddress.getLocalHost(InetAddress.java:1473) at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:59) at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:45) at kafka.server.KafkaServer.startup(KafkaServer.scala:121) at kafka.utils.TestUtils$.createServer(TestUtils.scala:130) at kafka.server.LogOffsetTest.setUp(LogOffsetTest.scala:53) Caused by: java.net.UnknownHostException: guwang-mn2: nodename nor servname provided, or not known at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293) at java.net.InetAddress.getLocalHost(InetAddress.java:1469) ... 5 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-724) Allow automatic socket.send.buffer from operating system
[ https://issues.apache.org/jira/browse/KAFKA-724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14332482#comment-14332482 ] ASF GitHub Bot commented on KAFKA-724: -- GitHub user rekhajoshm opened a pull request: https://github.com/apache/kafka/pull/50 KAFKA-724: auto socket buffer set KAFKA-724: Allow automatic socket.send.buffer from operating system If socket.receive.buffer.bytes/socket.send.buffer.bytes set to non-zero/-1, the OS defaults work.Do not explicitly set buffers. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rekhajoshm/kafka KAFKA-724 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/50.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #50 commit 118fdc3cdba2711d5d4389609da1b1fe759c5cab Author: Joshi rekhajo...@gmail.com Date: 2015-02-23T02:01:58Z KAFKA-724: auto socket buffer set Allow automatic socket.send.buffer from operating system Key: KAFKA-724 URL: https://issues.apache.org/jira/browse/KAFKA-724 Project: Kafka Issue Type: Improvement Reporter: Pablo Barrera Labels: newbie To do this, don't call to socket().setXXXBufferSize. This can be controlled by the configuration parameter: if the value socket.send.buffer or others are set to -1, don't call to socket().setXXXBufferSize -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1621) Standardize --messages option in perf scripts
[ https://issues.apache.org/jira/browse/KAFKA-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14514991#comment-14514991 ] ASF GitHub Bot commented on KAFKA-1621: --- GitHub user rekhajoshm opened a pull request: https://github.com/apache/kafka/pull/58 KAFKA-1621 : Standardize --messages option As per review comments from @nehanarkhede Thanks. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rekhajoshm/kafka localtrunk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/58.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #58 commit 51cab8584872457a349d6ba154a1087b9749e0f8 Author: Joshi rekhajo...@gmail.com Date: 2015-04-27T21:02:15Z KAFKA-1621 : Standardize --messages option Standardize --messages option in perf scripts - Key: KAFKA-1621 URL: https://issues.apache.org/jira/browse/KAFKA-1621 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1.1 Reporter: Jay Kreps Labels: newbie This option is specified in PerfConfig and is used by the producer, consumer and simple consumer perf commands. The docstring on the argument does not list it as required but the producer performance test requires it--others don't. We should standardize this so that either all the commands require the option and it is marked as required in the docstring or none of them list it as required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1054) Eliminate Compilation Warnings for 0.8 Final Release
[ https://issues.apache.org/jira/browse/KAFKA-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14513614#comment-14513614 ] ASF GitHub Bot commented on KAFKA-1054: --- GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/57 KAFKA-1054; Eliminate Scala Compilation Warnings Changes: - Suppressed compiler warnings about type erasure in matching via unboxing by Jon Riehl. - Suppressed warning caused by slight difference in input function type by John Riehl. - Fix compiler warnings: ServerShutdownTest, DelayedJoinGroup function signature by Blake Smith. - Fix Scala 2.11 warnings. `Pair` has been deprecated, `try` without `catch` and `finally` is useless and initialisation order fix by Ismael Juma. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka kafka-1054-squashed Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/57.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #57 commit 0a5501c84208ea2cb734cc2c2e30735503716f32 Author: Jon Riehl jri...@spaceship.com Date: 2014-09-05T23:25:32Z KAFKA-1054; Eliminate Scala Compilation Warnings Changes: - Suppressed compiler warnings about type erasure in matching via unboxing by Jon Riehl. - Suppressed warning caused by slight difference in input function type by John Riehl. - Fix compiler warnings: ServerShutdownTest, DelayedJoinGroup function signature by Blake Smith. - Fix Scala 2.11 warnings. `Pair` has been deprecated, `try` without `catch` and `finally` is useless and initialisation order fix by Ismael Juma. Eliminate Compilation Warnings for 0.8 Final Release Key: KAFKA-1054 URL: https://issues.apache.org/jira/browse/KAFKA-1054 Project: Kafka Issue Type: Improvement Reporter: Guozhang Wang Labels: newbie Fix For: 0.9.0 Attachments: KAFKA-1054-20150426-V1.patch, KAFKA-1054-20150426-V2.patch, KAFKA-1054.patch, KAFKA-1054_Mar_10_2015.patch Currently we have a total number of 38 warnings for source code compilation of 0.8. 1) 3 from Unchecked type pattern 2) 6 from Unchecked conversion 3) 29 from Deprecated Hadoop API functions It's better we finish these before the final release of 0.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2055) ConsumerBounceTest.testSeekAndCommitWithBrokerFailures transient failure
[
https://issues.apache.org/jira/browse/KAFKA-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523248#comment-14523248
]
ASF GitHub Bot commented on KAFKA-2055:
---
GitHub user lvfangmin opened a pull request:
https://github.com/apache/kafka/pull/60
Patch for KAFKA-2055: ConsumerBounceTest.testSeekAndCommitWithBrokerFail...
...ures transient failure
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lvfangmin/kafka KAFKA-2055
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/60.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #60
commit 5e174e3feabe03c14a6dca451fdf0967285069ff
Author: lvfangmin lvfang...@gmail.com
Date: 2015-04-27T14:40:35Z
Patch for KAFKA-2055:
ConsumerBounceTest.testSeekAndCommitWithBrokerFailures transient failure
ConsumerBounceTest.testSeekAndCommitWithBrokerFailures transient failure
Key: KAFKA-2055
URL: https://issues.apache.org/jira/browse/KAFKA-2055
Project: Kafka
Issue Type: Sub-task
Reporter: Guozhang Wang
Assignee: Fangmin Lv
Labels: newbie
Attachments: KAFKA-2055.patch
{code}
kafka.api.ConsumerBounceTest testSeekAndCommitWithBrokerFailures FAILED
java.lang.AssertionError: expected:1000 but was:976
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.failNotEquals(Assert.java:689)
at org.junit.Assert.assertEquals(Assert.java:127)
at org.junit.Assert.assertEquals(Assert.java:514)
at org.junit.Assert.assertEquals(Assert.java:498)
at
kafka.api.ConsumerBounceTest.seekAndCommitWithBrokerFailures(ConsumerBounceTest.scala:117)
at
kafka.api.ConsumerBounceTest.testSeekAndCommitWithBrokerFailures(ConsumerBounceTest.scala:98)
kafka.api.ConsumerBounceTest testSeekAndCommitWithBrokerFailures FAILED
java.lang.AssertionError: expected:1000 but was:913
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.failNotEquals(Assert.java:689)
at org.junit.Assert.assertEquals(Assert.java:127)
at org.junit.Assert.assertEquals(Assert.java:514)
at org.junit.Assert.assertEquals(Assert.java:498)
at
kafka.api.ConsumerBounceTest.seekAndCommitWithBrokerFailures(ConsumerBounceTest.scala:117)
at
kafka.api.ConsumerBounceTest.testSeekAndCommitWithBrokerFailures(ConsumerBounceTest.scala:98)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2169) Upgrade to zkclient-0.5
[ https://issues.apache.org/jira/browse/KAFKA-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533309#comment-14533309 ] ASF GitHub Bot commented on KAFKA-2169: --- GitHub user Parth-Brahmbhatt opened a pull request: https://github.com/apache/kafka/pull/61 KAFKA-2169: Moving to zkClient 0.5 release. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Parth-Brahmbhatt/kafka KAFKA-2169 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/61.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #61 commit e5eb373dcec7562292cec32f3962e42dda5cea24 Author: Parth Brahmbhatt brahmbhatt.pa...@gmail.com Date: 2015-05-07T20:15:55Z KAFKA-2169: Moving to zkClient 0.5 release. Upgrade to zkclient-0.5 --- Key: KAFKA-2169 URL: https://issues.apache.org/jira/browse/KAFKA-2169 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Neha Narkhede Assignee: Parth Brahmbhatt Priority: Critical zkclient-0.5 is released http://mvnrepository.com/artifact/com.101tec/zkclient/0.5 and has the fix for KAFKA-824 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2183) Add libvirt provider for vagrant. speed up vm for testing.
[ https://issues.apache.org/jira/browse/KAFKA-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537335#comment-14537335 ] ASF GitHub Bot commented on KAFKA-2183: --- GitHub user pronix opened a pull request: https://github.com/apache/kafka/pull/62 add support libvirt as provider. KAFKA-2183 https://issues.apache.org/jira/browse/KAFKA-2183 You can merge this pull request into a Git repository by running: $ git pull https://github.com/pronix/kafka add_libvirt_support Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/62.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #62 commit b70d79eb64eff2f2385779610217493343cd054a Author: dima pronix.serv...@gmail.com Date: 2015-05-10T19:39:35Z add support libvirt as provider Add libvirt provider for vagrant. speed up vm for testing. -- Key: KAFKA-2183 URL: https://issues.apache.org/jira/browse/KAFKA-2183 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.9.0 Environment: development environment Reporter: pronix Priority: Minor add libvirt provider for vagrant for testing and develop in virtual cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2167) ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)
[
https://issues.apache.org/jira/browse/KAFKA-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539267#comment-14539267
]
ASF GitHub Bot commented on KAFKA-2167:
---
GitHub user neeleshCloud opened a pull request:
https://github.com/apache/kafka/pull/63
Corrected the Changes in ZkUtils.scala - KAFKA-2167
Corrected Spelling errors.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/neeleshCloud/kafka 0.8.2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/63.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #63
commit 904af08e2437d2ddf57017a3f3f1decc2087c491
Author: Neelesh Srinivas Salian nsal...@cloudera.com
Date: 2015-05-12T05:17:42Z
KAFKA-2167
ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)
--
Key: KAFKA-2167
URL: https://issues.apache.org/jira/browse/KAFKA-2167
Project: Kafka
Issue Type: Bug
Reporter: Jon Bringhurst
Assignee: Neelesh Srinivas Salian
Labels: newbie
I'm not 100% sure on this, but it seems like persistent should instead say
ephemeral in the JavaDoc. Also, note that parrent is misspelled.
{noformat}
/**
* Update the value of a persistent node with the given path and data.
* create parrent directory if necessary. Never throw NodeExistException.
*/
def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit
= {
try {
client.writeData(path, data)
}
catch {
case e: ZkNoNodeException = {
createParentPath(client, path)
client.createEphemeral(path, data)
}
case e2 = throw e2
}
}
{noformat}
should be:
{noformat}
/**
* Update the value of an ephemeral node with the given path and data.
* create parent directory if necessary. Never throw NodeExistException.
*/
def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit
= {
try {
client.writeData(path, data)
}
catch {
case e: ZkNoNodeException = {
createParentPath(client, path)
client.createEphemeral(path, data)
}
case e2 = throw e2
}
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2167) ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)
[
https://issues.apache.org/jira/browse/KAFKA-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14540779#comment-14540779
]
ASF GitHub Bot commented on KAFKA-2167:
---
GitHub user nssalian opened a pull request:
https://github.com/apache/kafka/pull/64
Kafka 2167
Changed ZkUtils.scala for KAFKA-2167
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nssalian/kafka KAFKA-2167
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/64.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #64
commit 904af08e2437d2ddf57017a3f3f1decc2087c491
Author: Neelesh Srinivas Salian nsal...@cloudera.com
Date: 2015-05-12T05:17:42Z
KAFKA-2167
commit aa86ba400eab8d1511418ef7fea5f3d06db03b18
Author: Neelesh Srinivas Salian nsal...@cloudera.com
Date: 2015-05-12T21:09:01Z
Revert KAFKA-2167
This reverts commit 904af08e2437d2ddf57017a3f3f1decc2087c491.
commit 61cbb120e12c60fc952b2e2fdb96db5111e1551a
Author: Neelesh Srinivas Salian nsal...@cloudera.com
Date: 2015-05-12T21:20:06Z
Changed ZkUtils.scala
ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)
--
Key: KAFKA-2167
URL: https://issues.apache.org/jira/browse/KAFKA-2167
Project: Kafka
Issue Type: Bug
Reporter: Jon Bringhurst
Assignee: Neelesh Srinivas Salian
Labels: newbie
I'm not 100% sure on this, but it seems like persistent should instead say
ephemeral in the JavaDoc. Also, note that parrent is misspelled.
{noformat}
/**
* Update the value of a persistent node with the given path and data.
* create parrent directory if necessary. Never throw NodeExistException.
*/
def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit
= {
try {
client.writeData(path, data)
}
catch {
case e: ZkNoNodeException = {
createParentPath(client, path)
client.createEphemeral(path, data)
}
case e2 = throw e2
}
}
{noformat}
should be:
{noformat}
/**
* Update the value of an ephemeral node with the given path and data.
* create parent directory if necessary. Never throw NodeExistException.
*/
def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit
= {
try {
client.writeData(path, data)
}
catch {
case e: ZkNoNodeException = {
createParentPath(client, path)
client.createEphemeral(path, data)
}
case e2 = throw e2
}
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2167) ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)
[
https://issues.apache.org/jira/browse/KAFKA-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14540782#comment-14540782
]
ASF GitHub Bot commented on KAFKA-2167:
---
Github user nssalian closed the pull request at:
https://github.com/apache/kafka/pull/63
ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)
--
Key: KAFKA-2167
URL: https://issues.apache.org/jira/browse/KAFKA-2167
Project: Kafka
Issue Type: Bug
Reporter: Jon Bringhurst
Assignee: Neelesh Srinivas Salian
Labels: newbie
I'm not 100% sure on this, but it seems like persistent should instead say
ephemeral in the JavaDoc. Also, note that parrent is misspelled.
{noformat}
/**
* Update the value of a persistent node with the given path and data.
* create parrent directory if necessary. Never throw NodeExistException.
*/
def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit
= {
try {
client.writeData(path, data)
}
catch {
case e: ZkNoNodeException = {
createParentPath(client, path)
client.createEphemeral(path, data)
}
case e2 = throw e2
}
}
{noformat}
should be:
{noformat}
/**
* Update the value of an ephemeral node with the given path and data.
* create parent directory if necessary. Never throw NodeExistException.
*/
def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit
= {
try {
client.writeData(path, data)
}
catch {
case e: ZkNoNodeException = {
createParentPath(client, path)
client.createEphemeral(path, data)
}
case e2 = throw e2
}
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2098) Gradle Wrapper Jar gone missing in 0.8.2.1
[ https://issues.apache.org/jira/browse/KAFKA-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14481936#comment-14481936 ] ASF GitHub Bot commented on KAFKA-2098: --- GitHub user rekhajoshm opened a pull request: https://github.com/apache/kafka/pull/54 KAFKA-2098: gradle files gradle files, tiny footprint.lets have it in.thanks You can merge this pull request into a Git repository by running: $ git pull https://github.com/rekhajoshm/kafka KAFKA-2098 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/54.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #54 commit be7dc1933d2c266d15c46a06d057686c8d48e520 Author: Joshi rekhajo...@gmail.com Date: 2015-04-06T21:02:13Z KAFKA-2098: gradle files Gradle Wrapper Jar gone missing in 0.8.2.1 -- Key: KAFKA-2098 URL: https://issues.apache.org/jira/browse/KAFKA-2098 Project: Kafka Issue Type: Bug Components: build Affects Versions: 0.8.2.1 Reporter: Rekha Joshi ./gradlew idea Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain This was working in 0.8.2.Attaching patch.Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2289) KafkaProducer logs erroneous warning on startup
[ https://issues.apache.org/jira/browse/KAFKA-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594087#comment-14594087 ] ASF GitHub Bot commented on KAFKA-2289: --- GitHub user hgschmie opened a pull request: https://github.com/apache/kafka/pull/71 KAFKA-2289: KafkaProducer logs erroneous warning on startup This change fixes the problem. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hgschmie/kafka KAFKA-2289 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/71.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #71 commit ef6c0961c905d65ff8997a1ba7a09ded617b9837 Author: Henning Schmiedehausen henn...@groupon.com Date: 2015-06-19T23:17:38Z KAFKA-2289: KafkaProducer logs erroneous warning on startup This change fixes the problem. KafkaProducer logs erroneous warning on startup --- Key: KAFKA-2289 URL: https://issues.apache.org/jira/browse/KAFKA-2289 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.2.1 Reporter: Henning Schmiedehausen Priority: Trivial When creating a new KafkaProducer using the KafkaProducer(KafkaConfig, SerializerK, SerializerV) constructor, Kafka will list the following lines, which are harmless but are still at WARN level: WARN [2015-06-19 23:13:56,557] org.apache.kafka.clients.producer.ProducerConfig: The configuration value.serializer = class was supplied but isn't a known config. WARN [2015-06-19 23:13:56,557] org.apache.kafka.clients.producer.ProducerConfig: The configuration key.serializer = class was supplied but isn't a known config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2092) New partitioning for better load balancing
[ https://issues.apache.org/jira/browse/KAFKA-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14584688#comment-14584688 ] ASF GitHub Bot commented on KAFKA-2092: --- GitHub user gdfm opened a pull request: https://github.com/apache/kafka/pull/69 KAFKA-2092: New partitioning for better load balancing You can merge this pull request into a Git repository by running: $ git pull https://github.com/gdfm/kafka KAFKA-2092 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/69.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #69 commit ceaf4e3d4488e677ca584eca59d8d25ccb137f02 Author: Gianmarco De Francisci Morales g...@apache.org Date: 2015-06-13T15:59:15Z Add PKG partitioner for load balancing New partitioning for better load balancing -- Key: KAFKA-2092 URL: https://issues.apache.org/jira/browse/KAFKA-2092 Project: Kafka Issue Type: Improvement Components: producer Reporter: Gianmarco De Francisci Morales Assignee: Jun Rao Attachments: KAFKA-2092-v1.patch We have recently studied the problem of load balancing in distributed stream processing systems such as Samza [1]. In particular, we focused on what happens when the key distribution of the stream is skewed when using key grouping. We developed a new stream partitioning scheme (which we call Partial Key Grouping). It achieves better load balancing than hashing while being more scalable than round robin in terms of memory. In the paper we show a number of mining algorithms that are easy to implement with partial key grouping, and whose performance can benefit from it. We think that it might also be useful for a larger class of algorithms. PKG has already been integrated in Storm [2], and I would like to be able to use it in Samza as well. As far as I understand, Kafka producers are the ones that decide how to partition the stream (or Kafka topic). I do not have experience with Kafka, however partial key grouping is very easy to implement: it requires just a few lines of code in Java when implemented as a custom grouping in Storm [3]. I believe it should be very easy to integrate. For all these reasons, I believe it will be a nice addition to Kafka/Samza. If the community thinks it's a good idea, I will be happy to offer support in the porting. References: [1] https://melmeric.files.wordpress.com/2014/11/the-power-of-both-choices-practical-load-balancing-for-distributed-stream-processing-engines.pdf [2] https://issues.apache.org/jira/browse/STORM-632 [3] https://github.com/gdfm/partial-key-grouping -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2304) Support enabling JMX in Kafka Vagrantfile
[ https://issues.apache.org/jira/browse/KAFKA-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14617286#comment-14617286 ] ASF GitHub Bot commented on KAFKA-2304: --- Github user sslavic closed the pull request at: https://github.com/apache/kafka/pull/72 Support enabling JMX in Kafka Vagrantfile - Key: KAFKA-2304 URL: https://issues.apache.org/jira/browse/KAFKA-2304 Project: Kafka Issue Type: Bug Reporter: Stevo Slavic Assignee: Stevo Slavic Priority: Minor Fix For: 0.8.3 Attachments: KAFKA-2304-JMX.patch, KAFKA-2304-JMX.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1997) Refactor Mirror Maker
[ https://issues.apache.org/jira/browse/KAFKA-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662838#comment-14662838 ] ASF GitHub Bot commented on KAFKA-1997: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/120 Refactor Mirror Maker - Key: KAFKA-1997 URL: https://issues.apache.org/jira/browse/KAFKA-1997 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin Assignee: Jiangjie Qin Fix For: 0.8.3 Attachments: KAFKA-1997.patch, KAFKA-1997.patch, KAFKA-1997_2015-03-03_16:28:46.patch, KAFKA-1997_2015-03-04_15:07:46.patch, KAFKA-1997_2015-03-04_15:42:45.patch, KAFKA-1997_2015-03-05_20:14:58.patch, KAFKA-1997_2015-03-09_18:55:54.patch, KAFKA-1997_2015-03-10_18:31:34.patch, KAFKA-1997_2015-03-11_15:20:18.patch, KAFKA-1997_2015-03-11_19:10:53.patch, KAFKA-1997_2015-03-13_14:43:34.patch, KAFKA-1997_2015-03-17_13:47:01.patch, KAFKA-1997_2015-03-18_12:47:32.patch Refactor mirror maker based on KIP-3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2429) Add annotations to mark classes as stable/unstable
[ https://issues.apache.org/jira/browse/KAFKA-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694170#comment-14694170 ] ASF GitHub Bot commented on KAFKA-2429: --- GitHub user ewencp opened a pull request: https://github.com/apache/kafka/pull/133 KAFKA-2429: Add annotations to mark classes as stable/unstable This also marks the consumer as unstable to show an example of using these annotations. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ewencp/kafka stability-annotations Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/133.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #133 commit 09c15c37dcd128d608febbb9e578ef0ec85a471d Author: Ewen Cheslack-Postava m...@ewencp.org Date: 2015-08-12T21:04:01Z KAFKA-2429: Add annotations to mark classes as stable/unstable Add annotations to mark classes as stable/unstable -- Key: KAFKA-2429 URL: https://issues.apache.org/jira/browse/KAFKA-2429 Project: Kafka Issue Type: Improvement Reporter: Ewen Cheslack-Postava Assignee: Ewen Cheslack-Postava We should have some annotations so that we can mark classes as public and stable vs. in development and unstable. This will help address two issues. First, we already get fairly regular emails on the mailing list about non-functioning code because we sometimes check in stubbed out code to get started on some new code. Sometimes that also makes it into a release (e.g. the stubbed out interface for the new consumer). We don't expect that code to work, but it's not obvious to users that it shouldn't. Second, we sometimes want to be able to check in imperfect draft code because it's new, expected to be unstable, and it helps with reviewing to be able to get something smaller checked in and then iterate on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2429) Add annotations to mark classes as stable/unstable
[ https://issues.apache.org/jira/browse/KAFKA-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694249#comment-14694249 ] ASF GitHub Bot commented on KAFKA-2429: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/133 Add annotations to mark classes as stable/unstable -- Key: KAFKA-2429 URL: https://issues.apache.org/jira/browse/KAFKA-2429 Project: Kafka Issue Type: Improvement Reporter: Ewen Cheslack-Postava Assignee: Ewen Cheslack-Postava Fix For: 0.8.3 We should have some annotations so that we can mark classes as public and stable vs. in development and unstable. This will help address two issues. First, we already get fairly regular emails on the mailing list about non-functioning code because we sometimes check in stubbed out code to get started on some new code. Sometimes that also makes it into a release (e.g. the stubbed out interface for the new consumer). We don't expect that code to work, but it's not obvious to users that it shouldn't. Second, we sometimes want to be able to check in imperfect draft code because it's new, expected to be unstable, and it helps with reviewing to be able to get something smaller checked in and then iterate on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2300) Error in controller log when broker tries to rejoin cluster
[
https://issues.apache.org/jira/browse/KAFKA-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694208#comment-14694208
]
ASF GitHub Bot commented on KAFKA-2300:
---
Github user asfgit closed the pull request at:
https://github.com/apache/kafka/pull/102
Error in controller log when broker tries to rejoin cluster
---
Key: KAFKA-2300
URL: https://issues.apache.org/jira/browse/KAFKA-2300
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.2.1
Reporter: Johnny Brown
Assignee: Flavio Junqueira
Fix For: 0.8.3
Attachments: KAFKA-2300-controller-logs.tar.gz,
KAFKA-2300-repro.patch, KAFKA-2300.patch, KAFKA-2300.patch
Hello Kafka folks,
We are having an issue where a broker attempts to join the cluster after
being restarted, but is never added to the ISR for its assigned partitions.
This is a three-node cluster, and the controller is broker 2.
When broker 1 starts, we see the following message in broker 2's
controller.log.
{{
[2015-06-23 13:57:16,535] ERROR [BrokerChangeListener on Controller 2]: Error
while handling broker changes
(kafka.controller.ReplicaStateMachine$BrokerChangeListener)
java.lang.IllegalStateException: Controller to broker state change requests
batch is not empty while creating a new one. Some UpdateMetadata state
changes Map(2 - Map([prod-sver-end,1] -
(LeaderAndIsrInfo:(Leader:-2,ISR:1,LeaderEpoch:0,ControllerEpoch:165),ReplicationFactor:1),AllReplicas:1)),
1 - Map([prod-sver-end,1] -
(LeaderAndIsrInfo:(Leader:-2,ISR:1,LeaderEpoch:0,ControllerEpoch:165),ReplicationFactor:1),AllReplicas:1)),
3 - Map([prod-sver-end,1] -
(LeaderAndIsrInfo:(Leader:-2,ISR:1,LeaderEpoch:0,ControllerEpoch:165),ReplicationFactor:1),AllReplicas:1)))
might be lost
at
kafka.controller.ControllerBrokerRequestBatch.newBatch(ControllerChannelManager.scala:202)
at
kafka.controller.KafkaController.sendUpdateMetadataRequest(KafkaController.scala:974)
at
kafka.controller.KafkaController.onBrokerStartup(KafkaController.scala:399)
at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:371)
at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:358)
at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357)
at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357)
at kafka.utils.Utils$.inLock(Utils.scala:535)
at
kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:356)
at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
}}
{{prod-sver-end}} is a topic we previously deleted. It seems some remnant of
it persists in the controller's memory, causing an exception which interrupts
the state change triggered by the broker startup.
Has anyone seen something like this? Any idea what's happening here? Any
information would be greatly appreciated.
Thanks,
Johnny
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2389) CommitType seems not necessary in commit().
[ https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694519#comment-14694519 ] ASF GitHub Bot commented on KAFKA-2389: --- GitHub user becketqin opened a pull request: https://github.com/apache/kafka/pull/134 KAFKA-2389: remove commit type from new consumer. A shot to remove commit type from new consumer. The coordinator constructor takes a default offset commit callback mainly for testing purpose. You can merge this pull request into a Git repository by running: $ git pull https://github.com/becketqin/kafka KAFKA-2389 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/134.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #134 commit 5f332efa6c690fad730278283d8c419c6a223a8e Author: Jiangjie Qin becket@gmail.com Date: 2015-08-11T19:41:15Z KAFKA-2389: Remove commit type from commit() CommitType seems not necessary in commit(). --- Key: KAFKA-2389 URL: https://issues.apache.org/jira/browse/KAFKA-2389 Project: Kafka Issue Type: Sub-task Reporter: Jiangjie Qin Assignee: Jiangjie Qin The CommitType does not seem to be necessary in for commit(), it can be inferred from whether user passed in a callback or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2203) Get gradle build to work with Java 8
[ https://issues.apache.org/jira/browse/KAFKA-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700554#comment-14700554 ] ASF GitHub Bot commented on KAFKA-2203: --- GitHub user gwenshap opened a pull request: https://github.com/apache/kafka/pull/147 KAFKA-2203: Getting Java8 to relax about javadoc and let our build pass This patch is different than the one attached to the JIRA - I'm applying the new javadoc rules to all subprojects while the one in the JIRA applies only to clients. We need this since Copycat has the same issues. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gwenshap/kafka KAFKA-2203 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/147.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #147 commit 4f925d96457314bd42157dae8c8c40c5c08eda39 Author: Gwen Shapira csh...@gmail.com Date: 2015-08-18T01:05:13Z KAFKA-2203: Getting Java8 to relax about javadoc and let our build pass Get gradle build to work with Java 8 Key: KAFKA-2203 URL: https://issues.apache.org/jira/browse/KAFKA-2203 Project: Kafka Issue Type: Bug Components: build Affects Versions: 0.8.1.1 Reporter: Gaju Bhat Priority: Minor Fix For: 0.8.1.2 Attachments: 0001-Special-case-java-8-and-javadoc-handling.patch The gradle build halts because javadoc in java 8 is a lot stricter about valid html. It might be worthwhile to special case java 8 as described [here|http://blog.joda.org/2014/02/turning-off-doclint-in-jdk-8-javadoc.html]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2439) Add MirrorMakerService to ducktape system tests
[ https://issues.apache.org/jira/browse/KAFKA-2439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700909#comment-14700909 ] ASF GitHub Bot commented on KAFKA-2439: --- GitHub user granders opened a pull request: https://github.com/apache/kafka/pull/148 KAFKA-2439: Add MirrorMaker service class for system tests Added MirrorMaker service and a few corresponding sanity checks, as well as necessary config template files. A few additional updates to accomodate the change in wait_until from ducktape0.2.0-0.3.0 You can merge this pull request into a Git repository by running: $ git pull https://github.com/confluentinc/kafka KAFKA-2439 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/148.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #148 commit 1b4b04935eafa2e93583ea5683c2e8851ed43476 Author: Geoff Anderson ge...@confluent.io Date: 2015-08-18T08:15:00Z Added MirrorMaker service and a few corresponding sanity checks, as well as necessary config template files. A few additional updates to accomodate the change in wait_until from ducktape0.2.0-0.3.0 Add MirrorMakerService to ducktape system tests --- Key: KAFKA-2439 URL: https://issues.apache.org/jira/browse/KAFKA-2439 Project: Kafka Issue Type: Sub-task Components: system tests Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2411) remove usage of BlockingChannel in the broker
[ https://issues.apache.org/jira/browse/KAFKA-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703163#comment-14703163 ] ASF GitHub Bot commented on KAFKA-2411: --- GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/151 KAFKA-2411; remove usage of blocking channel You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka kafka-2411-remove-usage-of-blocking-channel Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/151.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #151 commit dbcde7e828a250708752866c4610298773dea006 Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-19T13:30:35Z Introduce `ChannelBuilders.create` and use it in `ClientUtils` and `SocketServer` commit 6de8b9b18c6bfb67e72a4fccc10768dff15098f8 Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-19T14:22:55Z Use `Selector` instead of `BlockingChannel` for controlled shutdown commit da7a980887ab2b5d007ddf80c3059b6619d52f99 Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-19T14:23:11Z Use `Selector` instead of `BlockingChannel` in `ControllerChannelManager` remove usage of BlockingChannel in the broker - Key: KAFKA-2411 URL: https://issues.apache.org/jira/browse/KAFKA-2411 Project: Kafka Issue Type: Sub-task Components: security Reporter: Jun Rao Assignee: Ismael Juma Fix For: 0.8.3 In KAFKA-1690, we are adding the SSL support at Selector. However, there are still a few places where we use BlockingChannel for inter-broker communication. We need to replace those usage with Selector/NetworkClient to enable inter-broker communication over SSL. Specially, BlockingChannel is currently used in the following places. 1. ControllerChannelManager: for the controller to propagate metadata to the brokers. 2. KafkaServer: for the broker to send controlled shutdown request to the controller. 3. -AbstractFetcherThread: for the follower to fetch data from the leader (through SimpleConsumer)- moved to KAFKA-2440 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2436) log.retention.hours should be honored by LogManager
[ https://issues.apache.org/jira/browse/KAFKA-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701894#comment-14701894 ] ASF GitHub Bot commented on KAFKA-2436: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/142 log.retention.hours should be honored by LogManager --- Key: KAFKA-2436 URL: https://issues.apache.org/jira/browse/KAFKA-2436 Project: Kafka Issue Type: Bug Reporter: Dong Lin Assignee: Dong Lin Priority: Critical Currently log.retention.hours is used to calculate KafkaConfig.logRetentionTimeMillis. But it is not used in LogManager to decide when to delete a log. LogManager is only using the log.retention.ms in the broker configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2446) KAFKA-2205 causes existing Topic config changes to be lost
[ https://issues.apache.org/jira/browse/KAFKA-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704033#comment-14704033 ] ASF GitHub Bot commented on KAFKA-2446: --- GitHub user auradkar opened a pull request: https://github.com/apache/kafka/pull/152 Fix for KAFKA-2446 This bug was introduced while committing KAFKA-2205. Basically, the path for topic overrides was renamed to topic from topics. However, this causes existing topic config overrides to break because they will not be read from ZK anymore since the path is different. https://reviews.apache.org/r/34554/ You can merge this pull request into a Git repository by running: $ git pull https://github.com/auradkar/kafka 2446 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/152.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #152 commit 4bb7adeb27673145dcb735f9e2039a05d94faea8 Author: Aditya Auradkar aaurad...@linkedin.com Date: 2015-08-20T00:12:04Z Fix for 2446 KAFKA-2205 causes existing Topic config changes to be lost -- Key: KAFKA-2446 URL: https://issues.apache.org/jira/browse/KAFKA-2446 Project: Kafka Issue Type: Bug Reporter: Aditya Auradkar Assignee: Aditya Auradkar The path was changed from /config/topics/ to /config/topic. This causes existing config overrides to not get read -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2446) KAFKA-2205 causes existing Topic config changes to be lost
[ https://issues.apache.org/jira/browse/KAFKA-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704103#comment-14704103 ] ASF GitHub Bot commented on KAFKA-2446: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/152 KAFKA-2205 causes existing Topic config changes to be lost -- Key: KAFKA-2446 URL: https://issues.apache.org/jira/browse/KAFKA-2446 Project: Kafka Issue Type: Bug Reporter: Aditya Auradkar Assignee: Aditya Auradkar The path was changed from /config/topics/ to /config/topic. This causes existing config overrides to not get read -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2457) StackOverflowError during builds
[ https://issues.apache.org/jira/browse/KAFKA-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706544#comment-14706544 ] ASF GitHub Bot commented on KAFKA-2457: --- GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/157 KAFKA-2457; StackOverflowError during builds The default is typically `1m` for 64-bit machines and the Scala compiler sometimes needs more than this. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka kafka-2457-stackoverflowerror-during-builds Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/157.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #157 commit 6054599e480f0f6b1fb413edd7a512766befba9b Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-21T10:53:46Z Set stack size at `2m` The default is typically `1m` for 64-bit machines and the Scala compiler sometimes needs more than this. StackOverflowError during builds Key: KAFKA-2457 URL: https://issues.apache.org/jira/browse/KAFKA-2457 Project: Kafka Issue Type: Bug Reporter: Ismael Juma Assignee: Ismael Juma Priority: Critical We need to set -Xss to avoid this problem. Will submit PR. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2411) remove usage of BlockingChannel in the broker
[ https://issues.apache.org/jira/browse/KAFKA-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707010#comment-14707010 ] ASF GitHub Bot commented on KAFKA-2411: --- Github user ijuma closed the pull request at: https://github.com/apache/kafka/pull/151 remove usage of BlockingChannel in the broker - Key: KAFKA-2411 URL: https://issues.apache.org/jira/browse/KAFKA-2411 Project: Kafka Issue Type: Sub-task Components: security Reporter: Jun Rao Assignee: Ismael Juma Fix For: 0.8.3 In KAFKA-1690, we are adding the SSL support at Selector. However, there are still a few places where we use BlockingChannel for inter-broker communication. We need to replace those usage with Selector/NetworkClient to enable inter-broker communication over SSL. Specially, BlockingChannel is currently used in the following places. 1. ControllerChannelManager: for the controller to propagate metadata to the brokers. 2. KafkaServer: for the broker to send controlled shutdown request to the controller. 3. -AbstractFetcherThread: for the follower to fetch data from the leader (through SimpleConsumer)- moved to KAFKA-2440 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2457) StackOverflowError during builds
[ https://issues.apache.org/jira/browse/KAFKA-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707327#comment-14707327 ] ASF GitHub Bot commented on KAFKA-2457: --- GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/159 KAFKA-2457; Fix how the argument is passed to `compileScala` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka kafka-2457-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/159.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #159 commit 562f9c18524a59aefa14fd7a46d213039ba47e46 Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-21T19:33:21Z KAFKA-2457; Fix how the argument is passed to `compileScala` StackOverflowError during builds Key: KAFKA-2457 URL: https://issues.apache.org/jira/browse/KAFKA-2457 Project: Kafka Issue Type: Bug Reporter: Ismael Juma Assignee: Ismael Juma Priority: Critical Fix For: 0.8.3 We need to set -Xss to avoid this problem. Will submit PR. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2457) StackOverflowError during builds
[ https://issues.apache.org/jira/browse/KAFKA-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707365#comment-14707365 ] ASF GitHub Bot commented on KAFKA-2457: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/159 StackOverflowError during builds Key: KAFKA-2457 URL: https://issues.apache.org/jira/browse/KAFKA-2457 Project: Kafka Issue Type: Bug Reporter: Ismael Juma Assignee: Ismael Juma Priority: Critical Fix For: 0.8.3 We need to set -Xss to avoid this problem. Will submit PR. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2457) StackOverflowError during builds
[ https://issues.apache.org/jira/browse/KAFKA-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707232#comment-14707232 ] ASF GitHub Bot commented on KAFKA-2457: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/157 StackOverflowError during builds Key: KAFKA-2457 URL: https://issues.apache.org/jira/browse/KAFKA-2457 Project: Kafka Issue Type: Bug Reporter: Ismael Juma Assignee: Ismael Juma Priority: Critical Fix For: 0.8.3 We need to set -Xss to avoid this problem. Will submit PR. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2442) QuotasTest should not fail when cpu is busy
[ https://issues.apache.org/jira/browse/KAFKA-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707491#comment-14707491 ] ASF GitHub Bot commented on KAFKA-2442: --- GitHub user auradkar opened a pull request: https://github.com/apache/kafka/pull/160 KAFKA-2442: Fixing transiently failing test Made the following changes: 1. Made the quotas very small. (100 and 10 bytes/sec for producer and consumer respectively) 2. For the producer, I'm asserting the throttle_time with a timed loop using waitUntilTrue 3. For the consumer, I'm simply calling a timed poll in a loop until the server side throttle time metric returns true You can merge this pull request into a Git repository by running: $ git pull https://github.com/auradkar/kafka failing-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/160.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #160 commit cd60472049dee0f515d4fc3f87bc1d6bba1eb923 Author: Aditya Auradkar aaurad...@linkedin.com Date: 2015-08-21T21:21:51Z KAFKA-2442: Fixing transiently failing test QuotasTest should not fail when cpu is busy --- Key: KAFKA-2442 URL: https://issues.apache.org/jira/browse/KAFKA-2442 Project: Kafka Issue Type: Bug Reporter: Dong Lin Assignee: Aditya Auradkar Fix For: 0.8.3 We observed that testThrottledProducerConsumer in QuotasTest may fail or succeed randomly. It appears that the test may fail when the system is slow. We can add timer in the integration test to avoid random failure. See an example failure at https://builds.apache.org/job/kafka-trunk-git-pr/166/console for patch https://github.com/apache/kafka/pull/142. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1543) Changing replication factor
[
https://issues.apache.org/jira/browse/KAFKA-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707713#comment-14707713
]
ASF GitHub Bot commented on KAFKA-1543:
---
GitHub user apakulov opened a pull request:
https://github.com/apache/kafka/pull/161
KAFKA-1543; Changing replication factor
Adding support to change replication-factor via kafka-topics to avoid
additional hassle of defining replicas explicilty. This change will allow to
make this change with one line:
```
kafka-topics.sh --zookeeper host:port --alter --topic name
--replication-factor 3
```
Also, made a small cleanup by replacing old junit.framework.Assert with
org.junit.Assert
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apakulov/kafka KAFKA-1543
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/161.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #161
commit ed276ffdac107afe26e82d051fbeb8fa4f67416f
Author: Alexander Pakulov a.paku...@gmail.com
Date: 2015-08-22T00:46:21Z
KAFKA-1543; Changing replication factor
Changing replication factor
---
Key: KAFKA-1543
URL: https://issues.apache.org/jira/browse/KAFKA-1543
Project: Kafka
Issue Type: Improvement
Reporter: Alexey Ozeritskiy
Assignee: Alexander Pakulov
Attachments: can-change-replication.patch
It is difficult to change replication factor by manual editing json config.
I propose to add a key to kafka-reassign-partitions.sh command to
automatically create json config.
Example of usage
{code}
kafka-reassign-partitions.sh --zookeeper zk --replicas new-replication-factor
--topics-to-move-json-file topics-file --broker-list 1,2,3,4 --generate
output
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2439) Add MirrorMakerService to ducktape system tests
[ https://issues.apache.org/jira/browse/KAFKA-2439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708271#comment-14708271 ] ASF GitHub Bot commented on KAFKA-2439: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/148 Add MirrorMakerService to ducktape system tests --- Key: KAFKA-2439 URL: https://issues.apache.org/jira/browse/KAFKA-2439 Project: Kafka Issue Type: Sub-task Components: system tests Reporter: Geoff Anderson Assignee: Geoff Anderson Labels: patch-available Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows
[ https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705922#comment-14705922 ] ASF GitHub Bot commented on KAFKA-2170: --- GitHub user mpoindexter opened a pull request: https://github.com/apache/kafka/pull/154 KAFKA-2170, KAFKA-1194: Fixes for Windows This branch fixes several Windows specific issues, both in the code and in the tests. With these changes the whole test suite passes on my Windows machine. I found the following issues that were relevant in Jira: KAFKA-2170 and KAFKA-1194, but there may be some others. I also have a branch with these changes done against 0.8.2.1 if there's any interest in merging to the 0.8 series. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mpoindexter/kafka fix-windows Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/154.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #154 commit 19ae8ac6a4c8ceef6451635055f75bd038fe25ae Author: Mike Poindexter statics...@gmail.com Date: 2015-08-20T02:01:25Z Fix Windows failures when renaming/deleting files - Windows will not allow a file that is mmap'd to be renamed or deleted. To work around this ensure that forceUnmap is called on close, delete and rename. For the rename case, make sure that the file is reopened after the rename completes - Windows will not allow a file that has an open FileChannel to be renamed. This causes breakage in FileMessageSet.renameTo since it holds the FileChannel open during rename. This can be worked around by changing how we open the FileChannel to use FileChannel.open instead of new FileInputStream(file).toChannel. This causes the file to be opened with the FILE_SHARE_DELETE flag which will allow the file to be renamed while open. See this JDK bug for details: http://bugs.java.com/view_bug.do?bug_id=6357433 - Fix a bug in LogTest that caused a race between the next iteration of a test loop and the asynchronous delete of old segments - Fix a bug in LogTest where the log was not closed leading to leftover garbage at the next run of a test loop - Ensure that any time forceUnmap is called we set mmap to null. This will ensure that invalid use after forceUnmap causes a NPE instead of JVM memory corruption commit 2066306285738d42be74c7987ee0ef91b8a6d7ee Author: Mike Poindexter statics...@gmail.com Date: 2015-08-20T22:26:14Z Fixes for load cleaning tests to ensure the files in a segment are only open once so renames, etc. do not fail on windows 10 LogTest cases failed for file.renameTo failed under windows --- Key: KAFKA-2170 URL: https://issues.apache.org/jira/browse/KAFKA-2170 Project: Kafka Issue Type: Bug Components: log Affects Versions: 0.9.0 Environment: Windows Reporter: Honghai Chen Assignee: Jay Kreps get latest code from trunk, then run test gradlew -i core:test --tests kafka.log.LogTest Got 10 cases failed for same reason: kafka.common.KafkaStorageException: Failed to change the log file suffix from to .deleted for log segment 0 at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:259) at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:756) at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:747) at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514) at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Log.deleteOldSegments(Log.scala:514) at kafka.log.LogTest.testAsyncDelete(LogTest.scala:633) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at
[jira] [Commented] (KAFKA-1566) Kafka environment configuration (kafka-env.sh)
[
https://issues.apache.org/jira/browse/KAFKA-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706128#comment-14706128
]
ASF GitHub Bot commented on KAFKA-1566:
---
GitHub user harshach opened a pull request:
https://github.com/apache/kafka/pull/156
KAFKA-1566: Kafka environment configuration (kafka-env.sh)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/harshach/kafka KAFKA-1566
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/156.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #156
commit 42b0acdb392494984b6928d94f0c611d4e1925de
Author: Sriharsha Chintalapani har...@hortonworks.com
Date: 2015-01-08T15:50:20Z
KAFKA-1566. Kafka environment configuration (kafka-env.sh).
commit 31d0dcab655b37864c207a08d5c77b9d27fff7bc
Author: Sriharsha Chintalapani har...@hortonworks.com
Date: 2015-03-18T00:14:41Z
KAFKA-1566. Kafka environment configuration (kafka-env.sh).
Kafka environment configuration (kafka-env.sh)
--
Key: KAFKA-1566
URL: https://issues.apache.org/jira/browse/KAFKA-1566
Project: Kafka
Issue Type: Improvement
Components: tools
Reporter: Cosmin Lehene
Assignee: Sriharsha Chintalapani
Labels: newbie
Fix For: 0.8.3
Attachments: KAFKA-1566.patch, KAFKA-1566_2015-02-21_21:57:02.patch,
KAFKA-1566_2015-03-17_17:01:38.patch, KAFKA-1566_2015-03-17_17:19:23.patch
It would be useful (especially for automated deployments) to have an
environment configuration file that could be sourced from the launcher files
(e.g. kafka-run-server.sh).
This is how this could look like kafka-env.sh
{code}
export KAFKA_JVM_PERFORMANCE_OPTS=-XX:+UseCompressedOops
-XX:+DisableExplicitGC -Djava.awt.headless=true \ -XX:+UseG1GC
-XX:PermSize=48m -XX:MaxPermSize=48m -XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35' %
export KAFKA_HEAP_OPTS='-Xmx1G -Xms1G' %
export KAFKA_LOG4J_OPTS=-Dkafka.logs.dir=/var/log/kafka
{code}
kafka-server-start.sh
{code}
...
source $base_dir/config/kafka-env.sh
...
{code}
This approach is consistent with Hadoop and HBase. However the idea here is
to be able to set these values in a single place without having to edit
startup scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706097#comment-14706097 ] ASF GitHub Bot commented on KAFKA-1683: --- GitHub user gwenshap opened a pull request: https://github.com/apache/kafka/pull/155 KAFKA-1683: persisting session information in Requests You can merge this pull request into a Git repository by running: $ git pull https://github.com/gwenshap/kafka KAFKA-1683 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/155.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #155 commit d97449d3626c1a209eefa2eb01e011ecdec4f147 Author: Gwen Shapira csh...@gmail.com Date: 2015-08-21T01:46:49Z persisting session information in Requests Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.9.0 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2454) Dead lock between delete log segment and shutting down.
[
https://issues.apache.org/jira/browse/KAFKA-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705491#comment-14705491
]
ASF GitHub Bot commented on KAFKA-2454:
---
GitHub user becketqin opened a pull request:
https://github.com/apache/kafka/pull/153
KAFKA-2454: Deadlock between log segment deletion and server shutdown.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/becketqin/kafka KAFKA-2454
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/153.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #153
commit 27c6a6e70f969ee13d857213d5a0aa3e40b1c19f
Author: Jiangjie Qin becket@gmail.com
Date: 2015-08-20T18:21:32Z
KAFKA-2454: Deadlock between log segment deletion and server shutdown.
Dead lock between delete log segment and shutting down.
---
Key: KAFKA-2454
URL: https://issues.apache.org/jira/browse/KAFKA-2454
Project: Kafka
Issue Type: Bug
Reporter: Jiangjie Qin
Assignee: Jiangjie Qin
When the broker shutdown, it will shutdown scheduler which grabs the
scheduler lock then wait for all the threads in scheduler to shutdown.
The dead lock will happen when the scheduled task try to delete old log
segment, it will schedule a log delete task which also needs to acquire the
scheduler lock. In this case the shutdown thread will hold scheduler lock and
wait for the the log deletion thread to finish, but the log deletion thread
will block on waiting for the scheduler lock.
Related stack trace:
{noformat}
Thread-1 #21 prio=5 os_prio=0 tid=0x7fe7601a7000 nid=0x1a4e waiting on
condition [0x7fe7cf698000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x000640d53540 (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1465)
at kafka.utils.KafkaScheduler.shutdown(KafkaScheduler.scala:94)
- locked 0x000640b6d480 (a kafka.utils.KafkaScheduler)
at
kafka.server.KafkaServer$$anonfun$shutdown$4.apply$mcV$sp(KafkaServer.scala:352)
at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:79)
at kafka.utils.Logging$class.swallowWarn(Logging.scala:92)
at kafka.utils.CoreUtils$.swallowWarn(CoreUtils.scala:51)
at kafka.utils.Logging$class.swallow(Logging.scala:94)
at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:51)
at kafka.server.KafkaServer.shutdown(KafkaServer.scala:352)
at
kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:42)
at com.linkedin.kafka.KafkaServer.notifyShutdown(KafkaServer.java:99)
at
com.linkedin.util.factory.lifecycle.LifeCycleMgr.notifyShutdownListener(LifeCycleMgr.java:123)
at
com.linkedin.util.factory.lifecycle.LifeCycleMgr.notifyListeners(LifeCycleMgr.java:102)
at
com.linkedin.util.factory.lifecycle.LifeCycleMgr.notifyStop(LifeCycleMgr.java:82)
- locked 0x000640b77bb0 (a java.util.ArrayDeque)
at com.linkedin.util.factory.Generator.stop(Generator.java:177)
- locked 0x000640b77bc8 (a java.lang.Object)
at
com.linkedin.offspring.servlet.OffspringServletRuntime.destroy(OffspringServletRuntime.java:82)
at
com.linkedin.offspring.servlet.OffspringServletContextListener.contextDestroyed(OffspringServletContextListener.java:51)
at
org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:813)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:160)
at
org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:516)
at com.linkedin.emweb.WebappContext.doStop(WebappContext.java:35)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
- locked 0x0006400018b8 (a java.lang.Object)
at
com.linkedin.emweb.ContextBasedHandlerImpl.doStop(ContextBasedHandlerImpl.java:112)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
- locked 0x000640001900 (a java.lang.Object)
at
com.linkedin.emweb.WebappDeployerImpl.stop(WebappDeployerImpl.java:349)
at
[jira] [Commented] (KAFKA-2434) remove roundrobin identical topic constraint in consumer coordinator (old API)
[ https://issues.apache.org/jira/browse/KAFKA-2434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699668#comment-14699668 ] ASF GitHub Bot commented on KAFKA-2434: --- GitHub user noslowerdna opened a pull request: https://github.com/apache/kafka/pull/145 KAFKA-2434: Remove identical topic subscription constraint for roundrobin strategy in old consumer API You can merge this pull request into a Git repository by running: $ git pull https://github.com/noslowerdna/kafka KAFKA-2434 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/145.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #145 commit d5b93efd9eef20f3f5836d779e6140d5e4a6fc65 Author: Andrew Olson aols...@cerner.com Date: 2015-08-17T15:01:44Z KAFKA-2434: Remove identical topic subscription constraint for roundrobin strategy in old consumer API remove roundrobin identical topic constraint in consumer coordinator (old API) -- Key: KAFKA-2434 URL: https://issues.apache.org/jira/browse/KAFKA-2434 Project: Kafka Issue Type: Sub-task Reporter: Andrew Olson Assignee: Andrew Olson Attachments: KAFKA-2434.patch The roundrobin strategy algorithm improvement made in KAFKA-2196 should be applied to the original high-level consumer API as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2435) More optimally balanced partition assignment strategy
[ https://issues.apache.org/jira/browse/KAFKA-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699669#comment-14699669 ] ASF GitHub Bot commented on KAFKA-2435: --- GitHub user noslowerdna opened a pull request: https://github.com/apache/kafka/pull/146 KAFKA-2435: Fair consumer partition assignment strategy You can merge this pull request into a Git repository by running: $ git pull https://github.com/noslowerdna/kafka KAFKA-2435 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/146.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #146 commit 080e265f742e1e23f84b86c7d99c45dce807d7c8 Author: Andrew Olson aols...@cerner.com Date: 2015-08-17T15:03:13Z KAFKA-2435: Fair consumer partition assignment strategy More optimally balanced partition assignment strategy - Key: KAFKA-2435 URL: https://issues.apache.org/jira/browse/KAFKA-2435 Project: Kafka Issue Type: Improvement Reporter: Andrew Olson Assignee: Andrew Olson Attachments: KAFKA-2435.patch While the roundrobin partition assignment strategy is an improvement over the range strategy, when the consumer topic subscriptions are not identical (previously disallowed but will be possible as of KAFKA-2172) it can produce heavily skewed assignments. As suggested [here|https://issues.apache.org/jira/browse/KAFKA-2172?focusedCommentId=14530767page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14530767] it would be nice to have a strategy that attempts to assign an equal number of partitions to each consumer in a group, regardless of how similar their individual topic subscriptions are. We can accomplish this by tracking the number of partitions assigned to each consumer, and having the partition assignment loop assign each partition to a consumer interested in that topic with the least number of partitions assigned. Additionally, we can optimize the distribution fairness by adjusting the partition assignment order: * Topics with fewer consumers are assigned first. * In the event of a tie for least consumers, the topic with more partitions is assigned first. The general idea behind these two rules is to keep the most flexible assignment choices available as long as possible by starting with the most constrained partitions/consumers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2377) Add copycat system tests
[ https://issues.apache.org/jira/browse/KAFKA-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702585#comment-14702585 ] ASF GitHub Bot commented on KAFKA-2377: --- GitHub user ewencp opened a pull request: https://github.com/apache/kafka/pull/150 KAFKA-2377: Add basic system test for copycat using source and sink file connectors. Tests standalone mode by running separate source and sink connectors, catting data into the source file, and validating the output in the sink file. Restarts the service to verify that clean restarts will result in tasks resuming where they left off. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ewencp/kafka kafka-2377-copycat-system-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/150.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #150 commit 10f6c17963b585c8430c491f836f13c09c00a5ec Author: Ewen Cheslack-Postava m...@ewencp.org Date: 2015-08-14T21:08:13Z KAFKA-2377: Add basic system test for copycat using source and sink file connectors. Tests standalone mode by running separate source and sink connectors, catting data into the source file, and validating the output in the sink file. Restarts the service to verify that clean restarts will result in tasks resuming where they left off. Add copycat system tests Key: KAFKA-2377 URL: https://issues.apache.org/jira/browse/KAFKA-2377 Project: Kafka Issue Type: Sub-task Components: copycat Reporter: Ewen Cheslack-Postava Fix For: 0.8.3 Add baseline system tests for Copycat, covering both standalone and distributed mode. This should cover basic failure modes and verify at-least-one delivery of data, both from source system - Kafka and Kafka - sink system. This, of course, requires testing the core, built-in connectors provided with Copycat. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2436) log.retention.hours should be honored by LogManager
[ https://issues.apache.org/jira/browse/KAFKA-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699986#comment-14699986 ] ASF GitHub Bot commented on KAFKA-2436: --- GitHub user lindong28 reopened a pull request: https://github.com/apache/kafka/pull/142 KAFKA-2436; log.retention.hours should be honored by LogManager You can merge this pull request into a Git repository by running: $ git pull https://github.com/lindong28/kafka KAFKA-2436 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/142.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #142 commit a713d45ad4ed59440be020cc6c74efbeb2bbe54b Author: Dong Lin lindon...@gmail.com Date: 2015-08-16T23:11:58Z KAFKA-2436; log.retention.hours should be honored by LogManager commit a8436f778bbdc498cdae741a74c33296065d0b21 Author: Dong Lin lindon...@gmail.com Date: 2015-08-17T15:30:14Z A few other configurations should also be propagated to LogManager commit ea44479abf305aa6926a1f6b2592c52216aa334f Author: Dong Lin lindon...@gmail.com Date: 2015-08-17T17:31:47Z remove unnecessary type conversion commit d634339fbb4daa0dfe112c6beea0751878613fa2 Author: Dong Lin lindon...@gmail.com Date: 2015-08-17T17:57:43Z move logRetentionTimeMillis to the group of Log Configuration in code log.retention.hours should be honored by LogManager --- Key: KAFKA-2436 URL: https://issues.apache.org/jira/browse/KAFKA-2436 Project: Kafka Issue Type: Bug Reporter: Dong Lin Assignee: Dong Lin Currently log.retention.hours is used to calculate KafkaConfig.logRetentionTimeMillis. But it is not used in LogManager to decide when to delete a log. LogManager is only using the log.retention.ms in the broker configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2436) log.retention.hours should be honored by LogManager
[ https://issues.apache.org/jira/browse/KAFKA-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699985#comment-14699985 ] ASF GitHub Bot commented on KAFKA-2436: --- Github user lindong28 closed the pull request at: https://github.com/apache/kafka/pull/142 log.retention.hours should be honored by LogManager --- Key: KAFKA-2436 URL: https://issues.apache.org/jira/browse/KAFKA-2436 Project: Kafka Issue Type: Bug Reporter: Dong Lin Assignee: Dong Lin Currently log.retention.hours is used to calculate KafkaConfig.logRetentionTimeMillis. But it is not used in LogManager to decide when to delete a log. LogManager is only using the log.retention.ms in the broker configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2436) log.retention.hours should be honored by LogManager
[ https://issues.apache.org/jira/browse/KAFKA-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698882#comment-14698882 ] ASF GitHub Bot commented on KAFKA-2436: --- GitHub user lindong28 opened a pull request: https://github.com/apache/kafka/pull/142 KAFKA-2436; log.retention.hours should be honored by LogManager You can merge this pull request into a Git repository by running: $ git pull https://github.com/lindong28/kafka KAFKA-2436 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/142.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #142 commit a713d45ad4ed59440be020cc6c74efbeb2bbe54b Author: Dong Lin lindon...@gmail.com Date: 2015-08-16T23:11:58Z KAFKA-2436; log.retention.hours should be honored by LogManager log.retention.hours should be honored by LogManager --- Key: KAFKA-2436 URL: https://issues.apache.org/jira/browse/KAFKA-2436 Project: Kafka Issue Type: Bug Reporter: Dong Lin Assignee: Dong Lin Currently log.retention.hours is used to calculate KafkaConfig.logRetentionTimeMillis. But it is not used in LogManager to decide when to delete a log. LogManager is only using the log.retention.ms in the broker configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2438) add maxParallelForks to build.gradle to speedup tests
[
https://issues.apache.org/jira/browse/KAFKA-2438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698967#comment-14698967
]
ASF GitHub Bot commented on KAFKA-2438:
---
GitHub user harshach opened a pull request:
https://github.com/apache/kafka/pull/143
KAFKA-2438: add maxParallelForks to build.gradle to speedup tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/harshach/kafka KAFKA-2438
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/143.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #143
commit 69b0edecc9becebc08c2a8a21777e8707fb3a564
Author: Sriharsha Chintalapani har...@hortonworks.com
Date: 2015-08-17T03:29:46Z
KAFKA-2438: add maxParallelForks to build.gradle to speedup tests.
add maxParallelForks to build.gradle to speedup tests
-
Key: KAFKA-2438
URL: https://issues.apache.org/jira/browse/KAFKA-2438
Project: Kafka
Issue Type: Bug
Reporter: Sriharsha Chintalapani
Assignee: Sriharsha Chintalapani
With current trunk unit tests on my machine takes 16+ mins and with this
patch runs about 6mins. Tested on OS X and linux.
{code}
BUILD SUCCESSFUL
Total time: 5 mins 37.194 secs
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2438) add maxParallelForks to build.gradle to speedup tests
[
https://issues.apache.org/jira/browse/KAFKA-2438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698982#comment-14698982
]
ASF GitHub Bot commented on KAFKA-2438:
---
Github user asfgit closed the pull request at:
https://github.com/apache/kafka/pull/143
add maxParallelForks to build.gradle to speedup tests
-
Key: KAFKA-2438
URL: https://issues.apache.org/jira/browse/KAFKA-2438
Project: Kafka
Issue Type: Bug
Reporter: Sriharsha Chintalapani
Assignee: Sriharsha Chintalapani
Fix For: 0.8.3
With current trunk unit tests on my machine takes 16+ mins and with this
patch runs about 6mins. Tested on OS X and linux.
Before
{code}
Total time: 18 mins 29.806 secs
{code}
After
{code}
BUILD SUCCESSFUL
Total time: 5 mins 37.194 secs
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2411) remove usage of BlockingChannel in the broker
[ https://issues.apache.org/jira/browse/KAFKA-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705145#comment-14705145 ] ASF GitHub Bot commented on KAFKA-2411: --- GitHub user ijuma reopened a pull request: https://github.com/apache/kafka/pull/151 KAFKA-2411; remove usage of blocking channel You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka kafka-2411-remove-usage-of-blocking-channel Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/151.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #151 commit dbcde7e828a250708752866c4610298773dea006 Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-19T13:30:35Z Introduce `ChannelBuilders.create` and use it in `ClientUtils` and `SocketServer` commit 6de8b9b18c6bfb67e72a4fccc10768dff15098f8 Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-19T14:22:55Z Use `Selector` instead of `BlockingChannel` for controlled shutdown commit da7a980887ab2b5d007ddf80c3059b6619d52f99 Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-19T14:23:11Z Use `Selector` instead of `BlockingChannel` in `ControllerChannelManager` commit 2b258901929e24fce2329bc85e650e4ca022bca0 Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-20T11:10:53Z Move `readCompletely` from `NetworkReceive` to `BlockingChannel` It is now a private method since it's not used anywhere else and it's been changed slightly to match the use-case better. commit f804f633d93d2beea94017bba9225504c2f9cea4 Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-20T12:29:55Z Adjust buffer and max request size to match `BlockingChannel` behaviour Based on feedback from Gwen. commit c71aab9b6e4c6172615a125d2406ff6f3d668996 Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-20T12:53:39Z Introduce specific methods in `SelectorUtils` and make the generic ones private As suggested by Gwen. commit 1de16166232e0b9a4b0798de493869d3ce23964c Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-20T13:17:45Z Reuse `Selector` when removing and re-adding brokers in `ControllerChannelManager` As suggested by Gwen. commit bf5b9c81fa59efa2429a409d3872f4f6f0d5d589 Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-20T14:55:03Z Merge remote-tracking branch 'apache/trunk' into kafka-2411-remove-usage-of-blocking-channel * apache/trunk: KAFKA-2330: Vagrantfile sets global configs instead of per-provider override configs; patched by Ewen Cheslack-Postava, reviewed by Geoff Anderson and Gwen Shapira KAFKA-2246; Fix incorrect config ZK path. KAFKA-2084; trivial follow-up (remove JUnit3Suite dependency) remove usage of BlockingChannel in the broker - Key: KAFKA-2411 URL: https://issues.apache.org/jira/browse/KAFKA-2411 Project: Kafka Issue Type: Sub-task Components: security Reporter: Jun Rao Assignee: Ismael Juma Fix For: 0.8.3 In KAFKA-1690, we are adding the SSL support at Selector. However, there are still a few places where we use BlockingChannel for inter-broker communication. We need to replace those usage with Selector/NetworkClient to enable inter-broker communication over SSL. Specially, BlockingChannel is currently used in the following places. 1. ControllerChannelManager: for the controller to propagate metadata to the brokers. 2. KafkaServer: for the broker to send controlled shutdown request to the controller. 3. -AbstractFetcherThread: for the follower to fetch data from the leader (through SimpleConsumer)- moved to KAFKA-2440 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2411) remove usage of BlockingChannel in the broker
[ https://issues.apache.org/jira/browse/KAFKA-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705144#comment-14705144 ] ASF GitHub Bot commented on KAFKA-2411: --- Github user ijuma closed the pull request at: https://github.com/apache/kafka/pull/151 remove usage of BlockingChannel in the broker - Key: KAFKA-2411 URL: https://issues.apache.org/jira/browse/KAFKA-2411 Project: Kafka Issue Type: Sub-task Components: security Reporter: Jun Rao Assignee: Ismael Juma Fix For: 0.8.3 In KAFKA-1690, we are adding the SSL support at Selector. However, there are still a few places where we use BlockingChannel for inter-broker communication. We need to replace those usage with Selector/NetworkClient to enable inter-broker communication over SSL. Specially, BlockingChannel is currently used in the following places. 1. ControllerChannelManager: for the controller to propagate metadata to the brokers. 2. KafkaServer: for the broker to send controlled shutdown request to the controller. 3. -AbstractFetcherThread: for the follower to fetch data from the leader (through SimpleConsumer)- moved to KAFKA-2440 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2015) Enable ConsoleConsumer to use new consumer
[ https://issues.apache.org/jira/browse/KAFKA-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705664#comment-14705664 ] ASF GitHub Bot commented on KAFKA-2015: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/144 Enable ConsoleConsumer to use new consumer -- Key: KAFKA-2015 URL: https://issues.apache.org/jira/browse/KAFKA-2015 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Guozhang Wang Assignee: Ben Stopford Fix For: 0.8.3 Attachments: KAFKA-2015.patch As titled, enable ConsoleConsumer to use new consumer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2276) Initial patch for KIP-25
[ https://issues.apache.org/jira/browse/KAFKA-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14645265#comment-14645265 ] ASF GitHub Bot commented on KAFKA-2276: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/70 Initial patch for KIP-25 Key: KAFKA-2276 URL: https://issues.apache.org/jira/browse/KAFKA-2276 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 Submit initial patch for KIP-25 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements) This patch should contain a few Service classes and a few tests which can serve as examples -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2350) Add KafkaConsumer pause capability
[
https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14645343#comment-14645343
]
ASF GitHub Bot commented on KAFKA-2350:
---
GitHub user hachikuji opened a pull request:
https://github.com/apache/kafka/pull/100
KAFKA-2350; KafkaConsumer pause/resume API
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hachikuji/kafka KAFKA-2350
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/100.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #100
commit a82b60a48f47a24b55d5b07ddb1a22fbcf52
Author: Jason Gustafson ja...@confluent.io
Date: 2015-07-29T00:58:10Z
KAFKA-2350; KafkaConsumer pause/resume API
Add KafkaConsumer pause capability
--
Key: KAFKA-2350
URL: https://issues.apache.org/jira/browse/KAFKA-2350
Project: Kafka
Issue Type: Improvement
Reporter: Jason Gustafson
Assignee: Jason Gustafson
There are some use cases in stream processing where it is helpful to be able
to pause consumption of a topic. For example, when joining two topics, you
may need to delay processing of one topic while you wait for the consumer of
the other topic to catch up. The new consumer currently doesn't provide a
nice way to do this. If you skip calls to poll() or if you unsubscribe, then
a rebalance will be triggered and your partitions will be reassigned to
another consumer. The desired behavior is instead that you keep the partition
assigned and simply
One way to achieve this would be to add two new methods to KafkaConsumer:
{code}
void pause(TopicPartition... partitions);
void resume(TopicPartition... partitions);
{code}
Here is the expected behavior of pause/resume:
* When a partition is paused, calls to KafkaConsumer.poll will not initiate
any new fetches for that partition.
* After the partition is resumed, fetches will begin again.
* While a partition is paused, seek() and position() can still be used to
advance or query the current position.
* Rebalance does not preserve pause/resume state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2300) Error in controller log when broker tries to rejoin cluster
[
https://issues.apache.org/jira/browse/KAFKA-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14646514#comment-14646514
]
ASF GitHub Bot commented on KAFKA-2300:
---
GitHub user fpj opened a pull request:
https://github.com/apache/kafka/pull/102
KAFKA-2300: Error in controller log when broker tries to rejoin cluster
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fpj/kafka 2300
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/102.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #102
commit dbd1bf3a91c3e15ed2d14bf941c41c87b8116608
Author: flavio junqueira f...@apache.org
Date: 2015-07-29T17:07:51Z
KAFKA-2300: Error in controller log when broker tries to rejoin cluster
Error in controller log when broker tries to rejoin cluster
---
Key: KAFKA-2300
URL: https://issues.apache.org/jira/browse/KAFKA-2300
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.2.1
Reporter: Johnny Brown
Assignee: Flavio Junqueira
Attachments: KAFKA-2300-controller-logs.tar.gz,
KAFKA-2300-repro.patch, KAFKA-2300.patch, KAFKA-2300.patch
Hello Kafka folks,
We are having an issue where a broker attempts to join the cluster after
being restarted, but is never added to the ISR for its assigned partitions.
This is a three-node cluster, and the controller is broker 2.
When broker 1 starts, we see the following message in broker 2's
controller.log.
{{
[2015-06-23 13:57:16,535] ERROR [BrokerChangeListener on Controller 2]: Error
while handling broker changes
(kafka.controller.ReplicaStateMachine$BrokerChangeListener)
java.lang.IllegalStateException: Controller to broker state change requests
batch is not empty while creating a new one. Some UpdateMetadata state
changes Map(2 - Map([prod-sver-end,1] -
(LeaderAndIsrInfo:(Leader:-2,ISR:1,LeaderEpoch:0,ControllerEpoch:165),ReplicationFactor:1),AllReplicas:1)),
1 - Map([prod-sver-end,1] -
(LeaderAndIsrInfo:(Leader:-2,ISR:1,LeaderEpoch:0,ControllerEpoch:165),ReplicationFactor:1),AllReplicas:1)),
3 - Map([prod-sver-end,1] -
(LeaderAndIsrInfo:(Leader:-2,ISR:1,LeaderEpoch:0,ControllerEpoch:165),ReplicationFactor:1),AllReplicas:1)))
might be lost
at
kafka.controller.ControllerBrokerRequestBatch.newBatch(ControllerChannelManager.scala:202)
at
kafka.controller.KafkaController.sendUpdateMetadataRequest(KafkaController.scala:974)
at
kafka.controller.KafkaController.onBrokerStartup(KafkaController.scala:399)
at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:371)
at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:358)
at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357)
at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357)
at kafka.utils.Utils$.inLock(Utils.scala:535)
at
kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:356)
at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
}}
{{prod-sver-end}} is a topic we previously deleted. It seems some remnant of
it persists in the controller's memory, causing an exception which interrupts
the state change triggered by the broker startup.
Has anyone seen something like this? Any idea what's happening here? Any
information would be greatly appreciated.
Thanks,
Johnny
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1371) Ignore build output dirs
[ https://issues.apache.org/jira/browse/KAFKA-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14646911#comment-14646911 ] ASF GitHub Bot commented on KAFKA-1371: --- Github user sslavic closed the pull request at: https://github.com/apache/kafka/pull/21 Ignore build output dirs Key: KAFKA-1371 URL: https://issues.apache.org/jira/browse/KAFKA-1371 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.8.1 Reporter: Stevo Slavic Assignee: Stevo Slavic Priority: Trivial Labels: git Fix For: 0.8.2.0 Attachments: 0001-Ignore-build-output-dirs.patch After a clean clone and project build, build output directories get reported as changes/new. They should be ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1370) Gradle startup script for Windows
[ https://issues.apache.org/jira/browse/KAFKA-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14646912#comment-14646912 ] ASF GitHub Bot commented on KAFKA-1370: --- Github user sslavic closed the pull request at: https://github.com/apache/kafka/pull/22 Gradle startup script for Windows - Key: KAFKA-1370 URL: https://issues.apache.org/jira/browse/KAFKA-1370 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.8.1 Reporter: Stevo Slavic Assignee: Stevo Slavic Priority: Trivial Labels: gradle Fix For: 0.8.2.0 Attachments: 0001-KAFKA-1370-Added-Gradle-startup-script-for-Windows.patch Please provide Gradle startup script for Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2397) leave group request
[ https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14650693#comment-14650693 ] ASF GitHub Bot commented on KAFKA-2397: --- GitHub user onurkaraman opened a pull request: https://github.com/apache/kafka/pull/103 KAFKA-2397: leave group request You can merge this pull request into a Git repository by running: $ git pull https://github.com/onurkaraman/kafka leave-group Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/103.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #103 commit 24d7c931f17f34211e3cac69a678ae0d3980396a Author: Onur Karaman okara...@linkedin.com Date: 2015-07-31T08:52:44Z leave group request leave group request --- Key: KAFKA-2397 URL: https://issues.apache.org/jira/browse/KAFKA-2397 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Onur Karaman Assignee: Onur Karaman Priority: Minor Fix For: 0.8.3 Let's say every consumer in a group has session timeout s. Currently, if a consumer leaves the group, the worst case time to stabilize the group is 2s (s to detect the consumer failure + s for the rebalance window). If a consumer instead can declare they are leaving the group, the worst case time to stabilize the group would just be the s associated with the rebalance window. This is a low priority optimization! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2386) Transient test failure: testGenerationIdIncrementsOnRebalance
[
https://issues.apache.org/jira/browse/KAFKA-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14652696#comment-14652696
]
ASF GitHub Bot commented on KAFKA-2386:
---
Github user asfgit closed the pull request at:
https://github.com/apache/kafka/pull/107
Transient test failure: testGenerationIdIncrementsOnRebalance
-
Key: KAFKA-2386
URL: https://issues.apache.org/jira/browse/KAFKA-2386
Project: Kafka
Issue Type: Sub-task
Reporter: Jason Gustafson
Seen this in some builds. Might be caused by gc pause (or similar) which
delays group join in the test.
{code}
kafka.coordinator.ConsumerCoordinatorResponseTest
testGenerationIdIncrementsOnRebalance FAILED
java.util.concurrent.TimeoutException: Futures timed out after [40
milliseconds]
at
scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at
scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at
kafka.coordinator.ConsumerCoordinatorResponseTest.joinGroup(ConsumerCoordinatorResponseTest.scala:313)
at
kafka.coordinator.ConsumerCoordinatorResponseTest.testGenerationIdIncrementsOnRebalance(ConsumerCoordinatorResponseTest.scala:272)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2386) Transient test failure: testGenerationIdIncrementsOnRebalance
[
https://issues.apache.org/jira/browse/KAFKA-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14652636#comment-14652636
]
ASF GitHub Bot commented on KAFKA-2386:
---
GitHub user hachikuji opened a pull request:
https://github.com/apache/kafka/pull/107
KAFKA-2386; increase timeouts for transient test failure in
ConsumerCoordinatorResponseTests
There are two race conditions in the test case
testGenerationIdIncrementsOnRebalance. First, a delay before the second join
group request can timeout the initial group and cause the generationId to
unexpectedly reset. Second, a delay in the join group request handling will
timeout the request itself and cause the test to fail. This commit doesn't
address these race conditions, but increases the timeouts to make them more
unlikely. If the problem reoccurs, then we'll probably need a better solution.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hachikuji/kafka KAFKA-2386
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/107.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #107
commit a53460a1a6637a814cbfe3731431746e56c52742
Author: Jason Gustafson ja...@confluent.io
Date: 2015-07-31T23:17:36Z
KAFKA-2386; increase timeouts for transient test failure in
ConsumerCoordinatorResponseTest
Transient test failure: testGenerationIdIncrementsOnRebalance
-
Key: KAFKA-2386
URL: https://issues.apache.org/jira/browse/KAFKA-2386
Project: Kafka
Issue Type: Sub-task
Reporter: Jason Gustafson
Seen this in some builds. Might be caused by gc pause (or similar) which
delays group join in the test.
{code}
kafka.coordinator.ConsumerCoordinatorResponseTest
testGenerationIdIncrementsOnRebalance FAILED
java.util.concurrent.TimeoutException: Futures timed out after [40
milliseconds]
at
scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at
scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at
kafka.coordinator.ConsumerCoordinatorResponseTest.joinGroup(ConsumerCoordinatorResponseTest.scala:313)
at
kafka.coordinator.ConsumerCoordinatorResponseTest.testGenerationIdIncrementsOnRebalance(ConsumerCoordinatorResponseTest.scala:272)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2350) Add KafkaConsumer pause capability
[
https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648329#comment-14648329
]
ASF GitHub Bot commented on KAFKA-2350:
---
Github user asfgit closed the pull request at:
https://github.com/apache/kafka/pull/100
Add KafkaConsumer pause capability
--
Key: KAFKA-2350
URL: https://issues.apache.org/jira/browse/KAFKA-2350
Project: Kafka
Issue Type: Improvement
Components: consumer
Reporter: Jason Gustafson
Assignee: Jason Gustafson
Fix For: 0.8.3
There are some use cases in stream processing where it is helpful to be able
to pause consumption of a topic. For example, when joining two topics, you
may need to delay processing of one topic while you wait for the consumer of
the other topic to catch up. The new consumer currently doesn't provide a
nice way to do this. If you skip calls to poll() or if you unsubscribe, then
a rebalance will be triggered and your partitions will be reassigned to
another consumer. The desired behavior is instead that you keep the partition
assigned and simply
One way to achieve this would be to add two new methods to KafkaConsumer:
{code}
void pause(TopicPartition... partitions);
void resume(TopicPartition... partitions);
{code}
Here is the expected behavior of pause/resume:
* When a partition is paused, calls to KafkaConsumer.poll will not initiate
any new fetches for that partition.
* After the partition is resumed, fetches will begin again.
* While a partition is paused, seek() and position() can still be used to
advance or query the current position.
* Rebalance does not preserve pause/resume state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2384) Override commit message title in kafka-merge-pr.py
[ https://issues.apache.org/jira/browse/KAFKA-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14653344#comment-14653344 ] ASF GitHub Bot commented on KAFKA-2384: --- GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/109 KAFKA-2384; Encode/decode to utf-8 for commit title IO in kafka-merge-pr.py This fix should be fine for Linux and OS X. Not sure about Windows though. This is a very specific fix for new functionality added in KAFKA-2384. There are other places where a similar error could occur, but are less likely. The script doesn't handle Unicode input properly in general. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka kafka-2384-hotfix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/109.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #109 commit 0ab8958846aba708a2d14e2e74d4707a34c7e725 Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-04T08:54:39Z Encode/decode to utf-8 for commit title IO in kafka-merge-pr.py Override commit message title in kafka-merge-pr.py -- Key: KAFKA-2384 URL: https://issues.apache.org/jira/browse/KAFKA-2384 Project: Kafka Issue Type: Improvement Reporter: Guozhang Wang Assignee: Ismael Juma Fix For: 0.8.3 It would be more convenient allow setting the commit message in the merging script; right now the script takes the PR title as is and the contributors have to change them according to the submission-review guidelines before doing the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2402) Broker should create zkpath /isr_change_notification if it does not exist when updating ISR.
[ https://issues.apache.org/jira/browse/KAFKA-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14653095#comment-14653095 ] ASF GitHub Bot commented on KAFKA-2402: --- GitHub user becketqin opened a pull request: https://github.com/apache/kafka/pull/108 KAFKA-2402: Create IsrChangeNotificationPath when server statrs. You can merge this pull request into a Git repository by running: $ git pull https://github.com/becketqin/kafka KAFKA-2402 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/108.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #108 commit 37e423066c5ff8a695a8fcde4f0c2e51832aa6b5 Author: Jiangjie Qin becket@gmail.com Date: 2015-08-04T05:36:20Z KAFKA-2402: Create IsrChangeNotificationPath when start the server. Broker should create zkpath /isr_change_notification if it does not exist when updating ISR. Key: KAFKA-2402 URL: https://issues.apache.org/jira/browse/KAFKA-2402 Project: Kafka Issue Type: Bug Reporter: Jiangjie Qin Assignee: Jiangjie Qin This is a follow up patch for KAFKA-1367. When broker update ISR of partitions, it should ensure zkPath /isr_change_notification exist. This does not matter when users do a clean deploy of Kafka cluster because controller will always create the cluster. But it matters when users are doing a rolling upgrade since the controller could still be running on a old version broker. In that case, ZkNoNodeException will be thrown and replica fetching will fail. We can either document the upgrade process to ask user create the zk path manually before upgrade or preferably we can handle it in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.
[ https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14654923#comment-14654923 ] ASF GitHub Bot commented on KAFKA-2406: --- GitHub user becketqin opened a pull request: https://github.com/apache/kafka/pull/114 KAFKA-2406[WIP]: Throttle ISR propagation This is a follow up patch for KAFKA-2406. Further test to verify if this change alone is enough to solve the problem or not. You can merge this pull request into a Git repository by running: $ git pull https://github.com/becketqin/kafka KAFKA-2406 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/114.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #114 commit 27224779c21781e87353a28d060322bfc2c70be2 Author: Jiangjie Qin becket@gmail.com Date: 2015-08-05T07:09:06Z KAFKA-2406: Throttle ISR propagation ISR propagation should be throttled to avoid overwhelming controller. - Key: KAFKA-2406 URL: https://issues.apache.org/jira/browse/KAFKA-2406 Project: Kafka Issue Type: Bug Reporter: Jiangjie Qin Assignee: Jiangjie Qin Priority: Blocker This is a follow up patch for KAFKA-1367. We need to throttle the ISR propagation rate to avoid flooding in controller to broker traffic. This might significantly increase time of controlled shutdown or cluster startup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2407) Only create a log directory when it will be used
[ https://issues.apache.org/jira/browse/KAFKA-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14658594#comment-14658594 ] ASF GitHub Bot commented on KAFKA-2407: --- GitHub user granthenke opened a pull request: https://github.com/apache/kafka/pull/115 KAFKA-2407: Only create log directory when it will be used You can merge this pull request into a Git repository by running: $ git pull https://github.com/granthenke/kafka log-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/115.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #115 commit 49a8dd4218b47978f0210c4e6ec0100aadaf3c21 Author: Grant Henke granthe...@gmail.com Date: 2015-08-05T17:37:07Z KAFKA-2407: Only create log directory when it will be used Only create a log directory when it will be used Key: KAFKA-2407 URL: https://issues.apache.org/jira/browse/KAFKA-2407 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Grant Henke Assignee: Grant Henke Fix For: 0.8.3 Currently kafka-run-class.sh will default the $LOG_DIR and create the directory regardless of it's use. This can cause permissions issues depending on what users are utilizing tools such as kafka-topics.sh. Further down in the script there is logic to detect whether $KAFKA_LOG4J_OPTS is set. If it is not set this is assumed to be a tool call and the script sets tools-log4j.properties which only uses the console appender. In this scenario a logging directory is not needed. In all other cases $KAFKA_LOG4J_OPTS will be set and we can move the $LOG_DIR defaulting creation logic there. For example kafka-server-start.sh sets $KAFKA_LOG4J_OPTS to use its own log4j.properties file which respects the $LOG_DIR/kafka.log4j.dir setting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2407) Only create a log directory when it will be used
[ https://issues.apache.org/jira/browse/KAFKA-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14658669#comment-14658669 ] ASF GitHub Bot commented on KAFKA-2407: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/115 Only create a log directory when it will be used Key: KAFKA-2407 URL: https://issues.apache.org/jira/browse/KAFKA-2407 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Grant Henke Assignee: Grant Henke Fix For: 0.8.3 Currently kafka-run-class.sh will default the $LOG_DIR and create the directory regardless of it's use. This can cause permissions issues depending on what users are utilizing tools such as kafka-topics.sh. Further down in the script there is logic to detect whether $KAFKA_LOG4J_OPTS is set. If it is not set this is assumed to be a tool call and the script sets tools-log4j.properties which only uses the console appender. In this scenario a logging directory is not needed. In all other cases $KAFKA_LOG4J_OPTS will be set and we can move the $LOG_DIR defaulting creation logic there. For example kafka-server-start.sh sets $KAFKA_LOG4J_OPTS to use its own log4j.properties file which respects the $LOG_DIR/kafka.log4j.dir setting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2400) Expose heartbeat frequency in new consumer configuration
[ https://issues.apache.org/jira/browse/KAFKA-2400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14658686#comment-14658686 ] ASF GitHub Bot commented on KAFKA-2400: --- GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/116 KAFKA-2400; expose heartbeat interval in KafkaConsumer configuration You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka KAFKA-2400 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/116.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #116 commit 3c1b1dd0dc44cd454d02aa7c476825c2ba46 Author: Jason Gustafson ja...@confluent.io Date: 2015-08-05T18:52:35Z KAFKA-2400; expose heartbeat interval in KafkaConsumer configuration Expose heartbeat frequency in new consumer configuration Key: KAFKA-2400 URL: https://issues.apache.org/jira/browse/KAFKA-2400 Project: Kafka Issue Type: Sub-task Reporter: Jason Gustafson Assignee: Jason Gustafson Priority: Minor The consumer coordinator communicates the need to rebalance through responses to heartbeat requests sent from each member of the consumer group. The heartbeat frequency therefore controls how long normal rebalances will take. Currently, the frequency is hard-coded to 3 heartbeats per the configured session timeout, but it would be nice to expose this setting so that the user can control the impact from rebalancing. Since the consumer is currently single-threaded and heartbeats are sent in poll(), we cannot guarantee that the heartbeats will actually be sent at the configured frequency. In practice, the user may have to adjust their fetch size to ensure that poll() is called often enough to get the desired heartbeat frequency. For most users, the consumption rate is probably fast enough for this not to matter, but we should make the documentation clear on this point. In any case, we expect that most users will accept the default value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2401) Fix transient failure of ProducerSendTest.testCloseWithZeroTimeoutFromSenderThread()
[ https://issues.apache.org/jira/browse/KAFKA-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14658755#comment-14658755 ] ASF GitHub Bot commented on KAFKA-2401: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/113 Fix transient failure of ProducerSendTest.testCloseWithZeroTimeoutFromSenderThread() Key: KAFKA-2401 URL: https://issues.apache.org/jira/browse/KAFKA-2401 Project: Kafka Issue Type: Sub-task Reporter: Jiangjie Qin Assignee: Jiangjie Qin Fix For: 0.8.3 The transient failure can happen because of a race condition of the callback firing order for messages produced to broker 0 and broker 1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2393) Correctly Handle InvalidTopicException in KafkaApis.getTopicMetadata()
[ https://issues.apache.org/jira/browse/KAFKA-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14659017#comment-14659017 ] ASF GitHub Bot commented on KAFKA-2393: --- GitHub user granthenke opened a pull request: https://github.com/apache/kafka/pull/117 KAFKA-2393: Correctly Handle InvalidTopicException in KafkaApis.getTo… …picMetadata() You can merge this pull request into a Git repository by running: $ git pull https://github.com/granthenke/kafka invalid-topic Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/117.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #117 commit 0abda5fffe7cb7cda585941e4909be304ad011f6 Author: Grant Henke granthe...@gmail.com Date: 2015-08-05T21:45:25Z KAFKA-2393: Correctly Handle InvalidTopicException in KafkaApis.getTopicMetadata() Correctly Handle InvalidTopicException in KafkaApis.getTopicMetadata() -- Key: KAFKA-2393 URL: https://issues.apache.org/jira/browse/KAFKA-2393 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Grant Henke Assignee: Grant Henke It seems that in KafkaApis.getTopicMetadata(), we need to handle InvalidTopicException explicitly when calling AdminUtils.createTopic (by returning the corresponding error code for that topic). Otherwise, we may not be able to get the metadata for other valid topics. This seems to be an existing problem, but KAFKA-2337 makes InvalidTopicException more likely to happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2340) Add additional unit tests for new consumer Fetcher
[ https://issues.apache.org/jira/browse/KAFKA-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14654596#comment-14654596 ] ASF GitHub Bot commented on KAFKA-2340: --- GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/112 KAFKA-2340; improve KafkaConsumer Fetcher test coverage You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka KAFKA-2340 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/112.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #112 commit d6424e566f20d521f535d0116af5bcd50ff1f249 Author: Jason Gustafson ja...@confluent.io Date: 2015-08-05T00:40:25Z KAFKA-2340; improve KafkaConsumer Fetcher test coverage Add additional unit tests for new consumer Fetcher -- Key: KAFKA-2340 URL: https://issues.apache.org/jira/browse/KAFKA-2340 Project: Kafka Issue Type: Test Reporter: Jason Gustafson There are a number of cases in Fetcher which have no corresponding unit tests. To name a few: - list offset with partition leader unknown - list offset disconnect - fetch disconnect Additionally, updateFetchPosition (which was moved from KafkaConsumer) has no tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2384) Override commit message title in kafka-merge-pr.py
[ https://issues.apache.org/jira/browse/KAFKA-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14651951#comment-14651951 ] ASF GitHub Bot commented on KAFKA-2384: --- GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/105 KAFKA-2384; Override commit message title in kafka-merge-pr.py You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka kafka-2384-override-commit-message-title Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/105.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #105 commit e04224273420ce4286a7f36356f99ce0df1af890 Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-03T14:23:05Z Support overriding of commit message title in kafka-merge-pr.py Override commit message title in kafka-merge-pr.py -- Key: KAFKA-2384 URL: https://issues.apache.org/jira/browse/KAFKA-2384 Project: Kafka Issue Type: Improvement Reporter: Guozhang Wang Assignee: Ismael Juma Fix For: 0.8.3 It would be more convenient allow setting the commit message in the merging script; right now the script takes the PR title as is and the contributors have to change them according to the submission-review guidelines before doing the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2399) Replace Stream.continually with Iterator.continually
[ https://issues.apache.org/jira/browse/KAFKA-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14652297#comment-14652297 ] ASF GitHub Bot commented on KAFKA-2399: --- GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/106 KAFKA-2399; Replace `Stream.continually` with `Iterator.continually` `Iterator.continually` is more efficient (it doesn't allocate a `Cons` instance per element) and we don't need the extra functionality provided by `Stream.continually`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka kafka-2399-replace-stream-continually Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/106.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #106 commit 68531ed09d4bf7f263e07c4d410dc1916d85666c Author: Ismael Juma ism...@juma.me.uk Date: 2015-08-03T18:54:04Z Replace `Stream.continually` with `Iterator.continually` Replace Stream.continually with Iterator.continually Key: KAFKA-2399 URL: https://issues.apache.org/jira/browse/KAFKA-2399 Project: Kafka Issue Type: Bug Reporter: Ismael Juma Priority: Minor There are two usages of `Stream.continually` and neither of them seems to need the extra functionality it provides over `Iterator.continually` (`Stream.continually` allocates `Cons` instances to save the computation instead of recomputing it if needed more than once). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2134) Producer blocked on metric publish
[ https://issues.apache.org/jira/browse/KAFKA-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14651343#comment-14651343 ] ASF GitHub Bot commented on KAFKA-2134: --- GitHub user becketqin opened a pull request: https://github.com/apache/kafka/pull/104 Fix for KAFKA-2134, fix replica offset truncate to beginning during leader migration. Fix replica truncate log to beginning during leader migration. You can merge this pull request into a Git repository by running: $ git pull https://github.com/becketqin/kafka KAFKA-2134 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/104.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #104 commit 71f8a4716e1f0b4fc2bd88aa30fe38aef8a9f92e Author: Jiangjie Qin becket@gmail.com Date: 2015-08-03T02:22:02Z Fix for KAFKA-2134, fix replica offset truncate to beginning during leader migration. Producer blocked on metric publish -- Key: KAFKA-2134 URL: https://issues.apache.org/jira/browse/KAFKA-2134 Project: Kafka Issue Type: Bug Components: producer Affects Versions: 0.8.2.1 Environment: debian7, java8 Reporter: Vamsi Subhash Achanta Assignee: Jun Rao Priority: Blocker Hi, We have a REST api to publish to a topic. Yesterday, we started noticing that the producer is not able to produce messages at a good rate and the CLOSE_WAITs of our producer REST app are very high. All the producer REST requests are hence timing out. When we took the thread dump and analysed it, we noticed that the threads are getting blocked on JmxReporter metricChange. Here is the attached stack trace. dw-70 - POST /queues/queue_1/messages #70 prio=5 os_prio=0 tid=0x7f043c8bd000 nid=0x54cf waiting for monitor entry [0x7f04363c7000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:76) - waiting to lock 0x0005c1823860 (a java.lang.Object) at org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:182) - locked 0x0007a5e526c8 (a org.apache.kafka.common.metrics.Metrics) at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:165) - locked 0x0007a5e526e8 (a org.apache.kafka.common.metrics.Sensor) When I looked at the code of metricChange method, it uses a synchronised block on an object resource and it seems that it is held by another. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2384) Override commit message title in kafka-merge-pr.py
[ https://issues.apache.org/jira/browse/KAFKA-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14652397#comment-14652397 ] ASF GitHub Bot commented on KAFKA-2384: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/105 Override commit message title in kafka-merge-pr.py -- Key: KAFKA-2384 URL: https://issues.apache.org/jira/browse/KAFKA-2384 Project: Kafka Issue Type: Improvement Reporter: Guozhang Wang Assignee: Ismael Juma Fix For: 0.8.3 It would be more convenient allow setting the commit message in the merging script; right now the script takes the PR title as is and the contributors have to change them according to the submission-review guidelines before doing the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1375) Formatting for Running a task on a particular version of Scala paragraph in README.md is broken
[ https://issues.apache.org/jira/browse/KAFKA-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14646907#comment-14646907 ] ASF GitHub Bot commented on KAFKA-1375: --- Github user sslavic closed the pull request at: https://github.com/apache/kafka/pull/24 Formatting for Running a task on a particular version of Scala paragraph in README.md is broken - Key: KAFKA-1375 URL: https://issues.apache.org/jira/browse/KAFKA-1375 Project: Kafka Issue Type: Bug Components: website Affects Versions: 0.8.1 Reporter: Stevo Slavic Assignee: Stevo Slavic Priority: Trivial Labels: documentation Fix For: 0.8.2.0 Attachments: 0001-KAFKA-1375-Fixed-formatting-of-instructions-for-usin.patch See commit which broke formatting at https://github.com/apache/kafka/commit/879e3e770ebc49f916137e8416df74373fa26a74#diff-04c6e90faac2675aa89e2176d2eec7d8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2413) New consumer's subscribe(Topic...) api fails if called more than once
[
https://issues.apache.org/jira/browse/KAFKA-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662548#comment-14662548
]
ASF GitHub Bot commented on KAFKA-2413:
---
Github user asfgit closed the pull request at:
https://github.com/apache/kafka/pull/122
New consumer's subscribe(Topic...) api fails if called more than once
-
Key: KAFKA-2413
URL: https://issues.apache.org/jira/browse/KAFKA-2413
Project: Kafka
Issue Type: Bug
Components: consumer
Reporter: Ashish K Singh
Assignee: Onur Karaman
Fix For: 0.8.3
I believe new consumer is supposed to allow adding to existing topic
subscriptions. If it is then the issue is that on trying to subscribe to a
topic when consumer is already subscribed to a topic, below exception is
thrown.
{code}
[2015-08-06 16:06:48,591] ERROR [KafkaApi-2] error when handling request null
(kafka.server.KafkaApis:103)
java.util.NoSuchElementException: key not found: topic
at scala.collection.MapLike$class.default(MapLike.scala:228)
at scala.collection.AbstractMap.default(Map.scala:58)
at scala.collection.MapLike$class.apply(MapLike.scala:141)
at scala.collection.AbstractMap.apply(Map.scala:58)
at
kafka.coordinator.RangeAssignor$$anonfun$4.apply(PartitionAssignor.scala:109)
at
kafka.coordinator.RangeAssignor$$anonfun$4.apply(PartitionAssignor.scala:108)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at kafka.coordinator.RangeAssignor.assign(PartitionAssignor.scala:108)
at
kafka.coordinator.ConsumerCoordinator.reassignPartitions(ConsumerCoordinator.scala:378)
at
kafka.coordinator.ConsumerCoordinator.rebalance(ConsumerCoordinator.scala:360)
at
kafka.coordinator.ConsumerCoordinator.onCompleteRebalance(ConsumerCoordinator.scala:414)
at
kafka.coordinator.DelayedRebalance.onComplete(DelayedRebalance.scala:39)
at
kafka.server.DelayedOperation.forceComplete(DelayedOperation.scala:72)
at
kafka.coordinator.DelayedRebalance$$anonfun$tryComplete$1.apply$mcZ$sp(DelayedRebalance.scala:37)
at
kafka.coordinator.ConsumerCoordinator.tryCompleteRebalance(ConsumerCoordinator.scala:388)
at
kafka.coordinator.DelayedRebalance.tryComplete(DelayedRebalance.scala:37)
at
kafka.server.DelayedOperationPurgatory$Watchers.tryCompleteWatched(DelayedOperation.scala:307)
at
kafka.server.DelayedOperationPurgatory.checkAndComplete(DelayedOperation.scala:227)
at
kafka.coordinator.ConsumerCoordinator.doJoinGroup(ConsumerCoordinator.scala:186)
at
kafka.coordinator.ConsumerCoordinator.handleJoinGroup(ConsumerCoordinator.scala:131)
at kafka.server.KafkaApis.handleJoinGroupRequest(KafkaApis.scala:578)
at kafka.server.KafkaApis.handle(KafkaApis.scala:67)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
Unexpected error in join group response: The server experienced an unexpected
error when processing the request
org.apache.kafka.common.KafkaException: Unexpected error in join group
response: The server experienced an unexpected error when processing the
request
at
org.apache.kafka.clients.consumer.internals.Coordinator$JoinGroupResponseHandler.handle(Coordinator.java:362)
at
org.apache.kafka.clients.consumer.internals.Coordinator$JoinGroupResponseHandler.handle(Coordinator.java:311)
at
org.apache.kafka.clients.consumer.internals.Coordinator$CoordinatorResponseHandler.onSuccess(Coordinator.java:703)
at
org.apache.kafka.clients.consumer.internals.Coordinator$CoordinatorResponseHandler.onSuccess(Coordinator.java:677)
at
org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:163)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:129)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:105)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:293)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:237)
at
[jira] [Commented] (KAFKA-1215) Rack-Aware replica assignment option
[ https://issues.apache.org/jira/browse/KAFKA-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14682185#comment-14682185 ] ASF GitHub Bot commented on KAFKA-1215: --- GitHub user allenxwang opened a pull request: https://github.com/apache/kafka/pull/132 KAFKA-1215: Rack-Aware replica assignment option The PR tries to achieve the following: - Make rack-aware assignment and rack data structure optional as opposed to be part of the core data structure/protocol to ease the migration. The implementation of that returns the map of broker to rack is pluggable. User needs to pass the implementation class as a Kafka runtime configuration or command line argument. - The rack aware replica assignment is best effort when distributing the replicas to racks. When there are more replicas than racks, it ensures each rack has at least one replica. When there are more racks than replicas, it ensures each rack has at most one replica. It also tries to keep the even distribution of replicas among brokers and racks when possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/allenxwang/kafka KAFKA-1215 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/132.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #132 commit 35db23ee7987a1811d630f14de66a99ce638 Author: Allen Wang aw...@netflix.com Date: 2015-08-11T17:52:37Z KAFKA-1215: Rack-Aware replica assignment option Rack-Aware replica assignment option Key: KAFKA-1215 URL: https://issues.apache.org/jira/browse/KAFKA-1215 Project: Kafka Issue Type: Improvement Components: replication Affects Versions: 0.8.0 Reporter: Joris Van Remoortere Assignee: Jun Rao Fix For: 0.9.0 Attachments: rack_aware_replica_assignment_v1.patch, rack_aware_replica_assignment_v2.patch Adding a rack-id to kafka config. This rack-id can be used during replica assignment by using the max-rack-replication argument in the admin scripts (create topic, etc.). By default the original replication assignment algorithm is used because max-rack-replication defaults to -1. max-rack-replication -1 is not honored if you are doing manual replica assignment (preffered). If this looks good I can add some test cases specific to the rack-aware assignment. I can also port this to trunk. We are currently running 0.8.0 in production and need this, so i wrote the patch against that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2408) (new) system tests: ConsoleConsumerService occasionally fails to register consumed message
[ https://issues.apache.org/jira/browse/KAFKA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14692379#comment-14692379 ] ASF GitHub Bot commented on KAFKA-2408: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/123 (new) system tests: ConsoleConsumerService occasionally fails to register consumed message -- Key: KAFKA-2408 URL: https://issues.apache.org/jira/browse/KAFKA-2408 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 There have been a few spurious failures in ReplicationTest.test_hard_bounce, where it was reported that a few of the acked messages were not consumed. Checking the logs, however, it is clear that they were consumed, but ConsoleConsumerService failed to parse. Lines causing parsing failure looks something like: 779725[2015-08-03 07:25:47,757] ERROR [ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1], Error for partition [test_topic,0] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) (i.e. the consumed message, and a log message appear on the same line) ConsoleConsumerService simply tries to strip each line of whitespace and parse as an integer, which will clearly fail in this case. Solution should either redirect stderr elsewhere or update parsing to handle this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2400) Expose heartbeat frequency in new consumer configuration
[ https://issues.apache.org/jira/browse/KAFKA-2400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660810#comment-14660810 ] ASF GitHub Bot commented on KAFKA-2400: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/116 Expose heartbeat frequency in new consumer configuration Key: KAFKA-2400 URL: https://issues.apache.org/jira/browse/KAFKA-2400 Project: Kafka Issue Type: Sub-task Reporter: Jason Gustafson Assignee: Jason Gustafson Priority: Minor Fix For: 0.8.3 The consumer coordinator communicates the need to rebalance through responses to heartbeat requests sent from each member of the consumer group. The heartbeat frequency therefore controls how long normal rebalances will take. Currently, the frequency is hard-coded to 3 heartbeats per the configured session timeout, but it would be nice to expose this setting so that the user can control the impact from rebalancing. Since the consumer is currently single-threaded and heartbeats are sent in poll(), we cannot guarantee that the heartbeats will actually be sent at the configured frequency. In practice, the user may have to adjust their fetch size to ensure that poll() is called often enough to get the desired heartbeat frequency. For most users, the consumption rate is probably fast enough for this not to matter, but we should make the documentation clear on this point. In any case, we expect that most users will accept the default value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2415) Transient failure in LogRecoveryTest.testHWCheckpointWithFailuresMultipleLogSegments
[ https://issues.apache.org/jira/browse/KAFKA-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661255#comment-14661255 ] ASF GitHub Bot commented on KAFKA-2415: --- GitHub user becketqin opened a pull request: https://github.com/apache/kafka/pull/121 KAFKA-2415: Fix transient failure in LogRecoveryTest You can merge this pull request into a Git repository by running: $ git pull https://github.com/becketqin/kafka KAFKA-2415 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/121.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #121 commit 346103c410f0afd875f95770b47a9235e0bf8ed2 Author: Jiangjie Qin j...@jqin-ld1.linkedin.biz Date: 2015-08-07T03:25:57Z KAFKA-2415: Fix transient failure in LogRecoveryTest Transient failure in LogRecoveryTest.testHWCheckpointWithFailuresMultipleLogSegments Key: KAFKA-2415 URL: https://issues.apache.org/jira/browse/KAFKA-2415 Project: Kafka Issue Type: Sub-task Reporter: Jiangjie Qin Assignee: Jiangjie Qin See transient failure in the test with the following error message. kafka.server.LogRecoveryTest testHWCheckpointWithFailuresMultipleLogSegments FAILED java.util.NoSuchElementException: None.get at scala.None$.get(Option.scala:313) at scala.None$.get(Option.scala:311) at kafka.server.LogRecoveryTest$$anonfun$testHWCheckpointWithFailuresMultipleLogSegments$8.apply$mcZ$sp(LogRecoveryTest.scala:215) at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:616) at kafka.server.LogRecoveryTest.testHWCheckpointWithFailuresMultipleLogSegments(LogRecoveryTest.scala:214) It looks the fix is to wait for the new broker to create the replica before check its HW. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2413) New consumer's subscribe(Topic...) api fails if called more than once
[
https://issues.apache.org/jira/browse/KAFKA-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661317#comment-14661317
]
ASF GitHub Bot commented on KAFKA-2413:
---
GitHub user onurkaraman opened a pull request:
https://github.com/apache/kafka/pull/122
KAFKA-2413: fix ConsumerCoordinator updateConsumer
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/onurkaraman/kafka KAFKA-2413
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/122.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #122
commit 073dc4b716594880de4fb58c8832f02dd3792683
Author: Onur Karaman okara...@linkedin.com
Date: 2015-08-07T04:49:53Z
fix ConsumerCoordinator updateConsumer
New consumer's subscribe(Topic...) api fails if called more than once
-
Key: KAFKA-2413
URL: https://issues.apache.org/jira/browse/KAFKA-2413
Project: Kafka
Issue Type: Bug
Components: consumer
Reporter: Ashish K Singh
Assignee: Ashish K Singh
I believe new consumer is supposed to allow adding to existing topic
subscriptions. If it is then the issue is that on trying to subscribe to a
topic when consumer is already subscribed to a topic, below exception is
thrown.
{code}
[2015-08-06 16:06:48,591] ERROR [KafkaApi-2] error when handling request null
(kafka.server.KafkaApis:103)
java.util.NoSuchElementException: key not found: topic
at scala.collection.MapLike$class.default(MapLike.scala:228)
at scala.collection.AbstractMap.default(Map.scala:58)
at scala.collection.MapLike$class.apply(MapLike.scala:141)
at scala.collection.AbstractMap.apply(Map.scala:58)
at
kafka.coordinator.RangeAssignor$$anonfun$4.apply(PartitionAssignor.scala:109)
at
kafka.coordinator.RangeAssignor$$anonfun$4.apply(PartitionAssignor.scala:108)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at kafka.coordinator.RangeAssignor.assign(PartitionAssignor.scala:108)
at
kafka.coordinator.ConsumerCoordinator.reassignPartitions(ConsumerCoordinator.scala:378)
at
kafka.coordinator.ConsumerCoordinator.rebalance(ConsumerCoordinator.scala:360)
at
kafka.coordinator.ConsumerCoordinator.onCompleteRebalance(ConsumerCoordinator.scala:414)
at
kafka.coordinator.DelayedRebalance.onComplete(DelayedRebalance.scala:39)
at
kafka.server.DelayedOperation.forceComplete(DelayedOperation.scala:72)
at
kafka.coordinator.DelayedRebalance$$anonfun$tryComplete$1.apply$mcZ$sp(DelayedRebalance.scala:37)
at
kafka.coordinator.ConsumerCoordinator.tryCompleteRebalance(ConsumerCoordinator.scala:388)
at
kafka.coordinator.DelayedRebalance.tryComplete(DelayedRebalance.scala:37)
at
kafka.server.DelayedOperationPurgatory$Watchers.tryCompleteWatched(DelayedOperation.scala:307)
at
kafka.server.DelayedOperationPurgatory.checkAndComplete(DelayedOperation.scala:227)
at
kafka.coordinator.ConsumerCoordinator.doJoinGroup(ConsumerCoordinator.scala:186)
at
kafka.coordinator.ConsumerCoordinator.handleJoinGroup(ConsumerCoordinator.scala:131)
at kafka.server.KafkaApis.handleJoinGroupRequest(KafkaApis.scala:578)
at kafka.server.KafkaApis.handle(KafkaApis.scala:67)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
Unexpected error in join group response: The server experienced an unexpected
error when processing the request
org.apache.kafka.common.KafkaException: Unexpected error in join group
response: The server experienced an unexpected error when processing the
request
at
org.apache.kafka.clients.consumer.internals.Coordinator$JoinGroupResponseHandler.handle(Coordinator.java:362)
at
org.apache.kafka.clients.consumer.internals.Coordinator$JoinGroupResponseHandler.handle(Coordinator.java:311)
at
org.apache.kafka.clients.consumer.internals.Coordinator$CoordinatorResponseHandler.onSuccess(Coordinator.java:703)
at
[jira] [Commented] (KAFKA-1997) Refactor Mirror Maker
[ https://issues.apache.org/jira/browse/KAFKA-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661250#comment-14661250 ] ASF GitHub Bot commented on KAFKA-1997: --- GitHub user becketqin opened a pull request: https://github.com/apache/kafka/pull/120 KAFKA-1997: Follow-up patch, hardcode key/value serializer in mirror maker to byte serializer. Hardcode the key/value serializer to ByteArraySerializer according to Jun’s comments. You can merge this pull request into a Git repository by running: $ git pull https://github.com/becketqin/kafka KAFKA-1997 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/120.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #120 commit 7f2e5a6ad9d43a4da8fdbcd5b2aaefa1de4c8963 Author: Jiangjie Qin j...@jqin-ld1.linkedin.biz Date: 2015-08-07T03:03:18Z KAFKA-1997: Follow-up patch, hardcode key/value serializer in mirror maker to byte serializer. Refactor Mirror Maker - Key: KAFKA-1997 URL: https://issues.apache.org/jira/browse/KAFKA-1997 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin Assignee: Jiangjie Qin Fix For: 0.8.3 Attachments: KAFKA-1997.patch, KAFKA-1997.patch, KAFKA-1997_2015-03-03_16:28:46.patch, KAFKA-1997_2015-03-04_15:07:46.patch, KAFKA-1997_2015-03-04_15:42:45.patch, KAFKA-1997_2015-03-05_20:14:58.patch, KAFKA-1997_2015-03-09_18:55:54.patch, KAFKA-1997_2015-03-10_18:31:34.patch, KAFKA-1997_2015-03-11_15:20:18.patch, KAFKA-1997_2015-03-11_19:10:53.patch, KAFKA-1997_2015-03-13_14:43:34.patch, KAFKA-1997_2015-03-17_13:47:01.patch, KAFKA-1997_2015-03-18_12:47:32.patch Refactor mirror maker based on KIP-3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2408) (new) system tests: ConsoleConsumerService occasionally fails to register consumed message
[ https://issues.apache.org/jira/browse/KAFKA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661403#comment-14661403 ] ASF GitHub Bot commented on KAFKA-2408: --- GitHub user granders opened a pull request: https://github.com/apache/kafka/pull/123 KAFKA-2408 ConsoleConsumerService direct log output to file You can merge this pull request into a Git repository by running: $ git pull https://github.com/confluentinc/kafka KAFKA-2408 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/123.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #123 commit 521a84b6d529d7f97cbb0e4cd099efdc5b88fe13 Author: Geoff Anderson ge...@confluent.io Date: 2015-08-07T01:36:26Z Updated console consumer to directo log output directly to file rather than stdout commit 8f890441aa755e3d79886b9d72bb572d3aa16fbd Author: Geoff Anderson ge...@confluent.io Date: 2015-08-07T02:44:49Z Added another lifecycle check. Wait for log file to exist before exmaning contents. commit af67e01bf0a114701117b069cfde0ba44a43c75c Author: Geoff Anderson ge...@confluent.io Date: 2015-08-07T06:08:55Z Merged in upstream trunk commit e67f55423b8da89ec8592fae314e59bd21e585ee Author: Geoff Anderson ge...@confluent.io Date: 2015-08-07T06:25:53Z Changed incorrect license header commit 66d6f4f2dc37e1d7f3dc348c1e4e2651a3fceaa1 Author: Geoff Anderson ge...@confluent.io Date: 2015-08-07T06:40:38Z lower - uperrcase constants (new) system tests: ConsoleConsumerService occasionally fails to register consumed message -- Key: KAFKA-2408 URL: https://issues.apache.org/jira/browse/KAFKA-2408 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 There have been a few spurious failures in ReplicationTest.test_hard_bounce, where it was reported that a few of the acked messages were not consumed. Checking the logs, however, it is clear that they were consumed, but ConsoleConsumerService failed to parse. Lines causing parsing failure looks something like: 779725[2015-08-03 07:25:47,757] ERROR [ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1], Error for partition [test_topic,0] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) (i.e. the consumed message, and a log message appear on the same line) ConsoleConsumerService simply tries to strip each line of whitespace and parse as an integer, which will clearly fail in this case. Solution should either redirect stderr elsewhere or update parsing to handle this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2405) KafkaHealthCheck kills the JVM in handleSessionEstablishmentError
[
https://issues.apache.org/jira/browse/KAFKA-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14654564#comment-14654564
]
ASF GitHub Bot commented on KAFKA-2405:
---
Github user asfgit closed the pull request at:
https://github.com/apache/kafka/pull/111
KafkaHealthCheck kills the JVM in handleSessionEstablishmentError
-
Key: KAFKA-2405
URL: https://issues.apache.org/jira/browse/KAFKA-2405
Project: Kafka
Issue Type: Bug
Components: core
Reporter: jaikiran pai
Assignee: jaikiran pai
Fix For: 0.8.3
The current code in KafkaHealthCheck in trunk does this:
{code}
override def handleSessionEstablishmentError(error: Throwable): Unit = {
fatal(Could not establish session with zookeeper, error)
System.exit(-1)
}
{code}
thus terminating the JVM. A session establishment error shouldn't cause the
JVM to terminate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2405) KafkaHealthCheck kills the JVM in handleSessionEstablishmentError
[
https://issues.apache.org/jira/browse/KAFKA-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14654481#comment-14654481
]
ASF GitHub Bot commented on KAFKA-2405:
---
GitHub user jaikiran opened a pull request:
https://github.com/apache/kafka/pull/111
KAFKA-2405 Don't kill the JVM on session establishment failure
As noted in the JIRA https://issues.apache.org/jira/browse/KAFKA-2405
currently the KafkaHealthCheck causes the JVM to terminate in cases where
session establishment with Zookeeper fails. I don't know if retrying (after a
while) is a better way to fix this but at least, IMO, the session establishment
failure shouldn't kill the JVM. This commit removes the `System.exit()` call.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jaikiran/kafka kafka-2405
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/111.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #111
commit f809162fcaf6cba9eddbc33dd4349ea453a1d8d8
Author: Jaikiran Pai jaikiran@gmail.com
Date: 2015-08-04T22:36:40Z
KAFKA-2405 Don't kill the JVM on session establishment failure
KafkaHealthCheck kills the JVM in handleSessionEstablishmentError
-
Key: KAFKA-2405
URL: https://issues.apache.org/jira/browse/KAFKA-2405
Project: Kafka
Issue Type: Bug
Components: core
Reporter: jaikiran pai
Assignee: jaikiran pai
The current code in KafkaHealthCheck in trunk does this:
{code}
override def handleSessionEstablishmentError(error: Throwable): Unit = {
fatal(Could not establish session with zookeeper, error)
System.exit(-1)
}
{code}
thus terminating the JVM. A session establishment error shouldn't cause the
JVM to terminate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2403) Expose offset commit metadata in new consumer
[
https://issues.apache.org/jira/browse/KAFKA-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660909#comment-14660909
]
ASF GitHub Bot commented on KAFKA-2403:
---
GitHub user hachikuji opened a pull request:
https://github.com/apache/kafka/pull/119
KAFKA-2403; Add support for commit metadata in KafkaConsumer
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hachikuji/kafka KAFKA-2403
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/119.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #119
commit 1ac0309682e170ca78495ade796b0f8957b04df9
Author: Jason Gustafson ja...@confluent.io
Date: 2015-08-05T22:10:02Z
KAFKA-2403; Add support for commit metadata in KafkaConsumer
Expose offset commit metadata in new consumer
-
Key: KAFKA-2403
URL: https://issues.apache.org/jira/browse/KAFKA-2403
Project: Kafka
Issue Type: Sub-task
Reporter: Jason Gustafson
Assignee: Jason Gustafson
Priority: Minor
The offset commit protocol supports the ability to add user metadata to
commits, but this is not yet exposed in the new consumer API. The
straightforward way to add it is to create a container for the offset and
metadata and adjust the KafkaConsumer API accordingly.
{code}
OffsetMetadata {
long offset;
String metadata;
}
KafkaConsumer {
commit(MapTopicPartition, OffsetMetadata)
OffsetMetadata committed(TopicPartition)
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
