[jira] [Created] (KAFKA-2780) kafka-server-start.sh globbing issue
Jon Bringhurst created KAFKA-2780:
-
Summary: kafka-server-start.sh globbing issue
Key: KAFKA-2780
URL: https://issues.apache.org/jira/browse/KAFKA-2780
Project: Kafka
Issue Type: Bug
Reporter: Jon Bringhurst
When spaces are present in the path leading up to kafka-server-start.sh,
kafka-server-start.sh has trouble finding kafka-run-class.sh.
The path that is executed should be quoted.
{noformat}
jbringhu@jbringhu-mn3 /V/N/L/t/t/m/M/kafka_2.11-0.8.2.2> /Volumes/NO\
NAME/LISA\
15/training_materials/training-program/materials/M4/kafka_2.11-0.8.2.2/bin/kafka-server-start.sh
./config/server.properties
usage: dirname path
/Volumes/NO NAME/LISA
15/training_materials/training-program/materials/M4/kafka_2.11-0.8.2.2/bin/kafka-server-start.sh:
line 44: /kafka-run-class.sh: No such file or directory
/Volumes/NO NAME/LISA
15/training_materials/training-program/materials/M4/kafka_2.11-0.8.2.2/bin/kafka-server-start.sh:
line 44: exec: /kafka-run-class.sh: cannot execute: No such file or directory
jbringhu@jbringhu-mn3 /V/N/L/t/t/m/M/kafka_2.11-0.8.2.2>
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2780) kafka-server-start.sh globbing issue
[
https://issues.apache.org/jira/browse/KAFKA-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jon Bringhurst updated KAFKA-2780:
--
Description:
When spaces are present in the path leading up to kafka-server-start.sh,
kafka-server-start.sh has trouble finding kafka-run-class.sh.
The path that is executed should be quoted.
{noformat}
jbringhu@jbringhu-mn3 /V/N/L/t/t/m/M/kafka_2.11-0.8.2.2> /Volumes/NO\
NAME/LISA\
15/training_materials/training-program/materials/M4/kafka_2.11-0.8.2.2/bin/kafka-server-start.sh
./config/server.properties
usage: dirname path
/Volumes/NO NAME/LISA
15/training_materials/training-program/materials/M4/kafka_2.11-0.8.2.2/bin/kafka-server-start.sh:
line 44: /kafka-run-class.sh: No such file or directory
/Volumes/NO NAME/LISA
15/training_materials/training-program/materials/M4/kafka_2.11-0.8.2.2/bin/kafka-server-start.sh:
line 44: exec: /kafka-run-class.sh: cannot execute: No such file or directory
jbringhu@jbringhu-mn3 /V/N/L/t/t/m/M/kafka_2.11-0.8.2.2>
{noformat}
This was discovered during LISA15. After I'm back from the conference, I'll try
to edit this and add simple instructions to reproduce (and maybe a patch).
was:
When spaces are present in the path leading up to kafka-server-start.sh,
kafka-server-start.sh has trouble finding kafka-run-class.sh.
The path that is executed should be quoted.
{noformat}
jbringhu@jbringhu-mn3 /V/N/L/t/t/m/M/kafka_2.11-0.8.2.2> /Volumes/NO\
NAME/LISA\
15/training_materials/training-program/materials/M4/kafka_2.11-0.8.2.2/bin/kafka-server-start.sh
./config/server.properties
usage: dirname path
/Volumes/NO NAME/LISA
15/training_materials/training-program/materials/M4/kafka_2.11-0.8.2.2/bin/kafka-server-start.sh:
line 44: /kafka-run-class.sh: No such file or directory
/Volumes/NO NAME/LISA
15/training_materials/training-program/materials/M4/kafka_2.11-0.8.2.2/bin/kafka-server-start.sh:
line 44: exec: /kafka-run-class.sh: cannot execute: No such file or directory
jbringhu@jbringhu-mn3 /V/N/L/t/t/m/M/kafka_2.11-0.8.2.2>
{noformat}
> kafka-server-start.sh globbing issue
>
>
> Key: KAFKA-2780
> URL: https://issues.apache.org/jira/browse/KAFKA-2780
> Project: Kafka
> Issue Type: Bug
>Reporter: Jon Bringhurst
>
> When spaces are present in the path leading up to kafka-server-start.sh,
> kafka-server-start.sh has trouble finding kafka-run-class.sh.
> The path that is executed should be quoted.
> {noformat}
> jbringhu@jbringhu-mn3 /V/N/L/t/t/m/M/kafka_2.11-0.8.2.2> /Volumes/NO\
> NAME/LISA\
> 15/training_materials/training-program/materials/M4/kafka_2.11-0.8.2.2/bin/kafka-server-start.sh
> ./config/server.properties
> usage: dirname path
> /Volumes/NO NAME/LISA
> 15/training_materials/training-program/materials/M4/kafka_2.11-0.8.2.2/bin/kafka-server-start.sh:
> line 44: /kafka-run-class.sh: No such file or directory
> /Volumes/NO NAME/LISA
> 15/training_materials/training-program/materials/M4/kafka_2.11-0.8.2.2/bin/kafka-server-start.sh:
> line 44: exec: /kafka-run-class.sh: cannot execute: No such file or directory
> jbringhu@jbringhu-mn3 /V/N/L/t/t/m/M/kafka_2.11-0.8.2.2>
> {noformat}
> This was discovered during LISA15. After I'm back from the conference, I'll
> try to edit this and add simple instructions to reproduce (and maybe a patch).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KAFKA-2167) ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)
Jon Bringhurst created KAFKA-2167:
-
Summary: ZkUtils updateEphemeralPath JavaDoc (spelling and
correctness)
Key: KAFKA-2167
URL: https://issues.apache.org/jira/browse/KAFKA-2167
Project: Kafka
Issue Type: Bug
Reporter: Jon Bringhurst
I'm not 100% sure on this, but it seems like persistent should instead say
ephemeral in the JavaDoc. Also, note that parrent is misspelled.
{noformat}
/**
* Update the value of a persistent node with the given path and data.
* create parrent directory if necessary. Never throw NodeExistException.
*/
def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit =
{
try {
client.writeData(path, data)
}
catch {
case e: ZkNoNodeException = {
createParentPath(client, path)
client.createEphemeral(path, data)
}
case e2 = throw e2
}
}
{noformat}
should be:
{noformat}
/**
* Update the value of an ephemeral node with the given path and data.
* create parent directory if necessary. Never throw NodeExistException.
*/
def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit =
{
try {
client.writeData(path, data)
}
catch {
case e: ZkNoNodeException = {
createParentPath(client, path)
client.createEphemeral(path, data)
}
case e2 = throw e2
}
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2167) ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)
[
https://issues.apache.org/jira/browse/KAFKA-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jon Bringhurst updated KAFKA-2167:
--
Labels: newbie (was: )
ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)
--
Key: KAFKA-2167
URL: https://issues.apache.org/jira/browse/KAFKA-2167
Project: Kafka
Issue Type: Bug
Reporter: Jon Bringhurst
Labels: newbie
I'm not 100% sure on this, but it seems like persistent should instead say
ephemeral in the JavaDoc. Also, note that parrent is misspelled.
{noformat}
/**
* Update the value of a persistent node with the given path and data.
* create parrent directory if necessary. Never throw NodeExistException.
*/
def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit
= {
try {
client.writeData(path, data)
}
catch {
case e: ZkNoNodeException = {
createParentPath(client, path)
client.createEphemeral(path, data)
}
case e2 = throw e2
}
}
{noformat}
should be:
{noformat}
/**
* Update the value of an ephemeral node with the given path and data.
* create parent directory if necessary. Never throw NodeExistException.
*/
def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit
= {
try {
client.writeData(path, data)
}
catch {
case e: ZkNoNodeException = {
createParentPath(client, path)
client.createEphemeral(path, data)
}
case e2 = throw e2
}
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494808#comment-14494808 ] Jon Bringhurst commented on KAFKA-1367: --- Hey [~jkreps], I can confirm that the metadata response is still out of sync with ZK in a recent trunk build (about a month old). Btw, it's not much effort for me to simply look at ZK or JMX -- it was just confusing when I initially ran into this. Broker topic metadata not kept in sync with ZooKeeper - Key: KAFKA-1367 URL: https://issues.apache.org/jira/browse/KAFKA-1367 Project: Kafka Issue Type: Bug Affects Versions: 0.8.0, 0.8.1 Reporter: Ryan Berdeen Labels: newbie++ Fix For: 0.8.3 Attachments: KAFKA-1367.txt When a broker is restarted, the topic metadata responses from the brokers will be incorrect (different from ZooKeeper) until a preferred replica leader election. In the metadata, it looks like leaders are correctly removed from the ISR when a broker disappears, but followers are not. Then, when a broker reappears, the ISR is never updated. I used a variation of the Vagrant setup created by Joe Stein to reproduce this with latest from the 0.8.1 branch: https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1641) Log cleaner exits if last cleaned offset is lower than earliest offset
[
https://issues.apache.org/jira/browse/KAFKA-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140887#comment-14140887
]
Jon Bringhurst commented on KAFKA-1641:
---
I'd just like to mention that a possible workaround (depending on your
situation) is to stop the broker, remove the cleaner offset checkpoint, and
then start the broker again for each ISR member in serial to get the thread
running again. Keep in mind that the cleaner will start from the beginning if
you do this.
Log cleaner exits if last cleaned offset is lower than earliest offset
--
Key: KAFKA-1641
URL: https://issues.apache.org/jira/browse/KAFKA-1641
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Joel Koshy
Encountered this recently: the log cleaner exited a while ago (I think
because the topic had compressed messages). That issue was subsequently
addressed by having the producer only send uncompressed. However, on a
subsequent restart of the broker we see this:
In this scenario I think it is reasonable to just emit a warning and have the
cleaner round up its first dirty offset to the base offset of the first
segment.
{code}
[kafka-server] [] [kafka-log-cleaner-thread-0], Error due to
java.lang.IllegalArgumentException: requirement failed: Last clean offset is
54770438 but segment base offset is 382844024 for log testtopic-0.
at scala.Predef$.require(Predef.scala:145)
at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:491)
at kafka.log.Cleaner.clean(LogCleaner.scala:288)
at
kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:202)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:187)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1641) Log cleaner exits if last cleaned offset is lower than earliest offset
[
https://issues.apache.org/jira/browse/KAFKA-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140887#comment-14140887
]
Jon Bringhurst edited comment on KAFKA-1641 at 9/19/14 5:10 PM:
I'd just like to mention that a possible workaround (depending on your
situation in regard to keys) is to stop the broker, remove the cleaner offset
checkpoint, and then start the broker again for each ISR member in serial to
get the thread running again. Keep in mind that the cleaner will start from the
beginning if you do this.
was (Author: jonbringhurst):
I'd just like to mention that a possible workaround (depending on your
situation) is to stop the broker, remove the cleaner offset checkpoint, and
then start the broker again for each ISR member in serial to get the thread
running again. Keep in mind that the cleaner will start from the beginning if
you do this.
Log cleaner exits if last cleaned offset is lower than earliest offset
--
Key: KAFKA-1641
URL: https://issues.apache.org/jira/browse/KAFKA-1641
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Joel Koshy
Encountered this recently: the log cleaner exited a while ago (I think
because the topic had compressed messages). That issue was subsequently
addressed by having the producer only send uncompressed. However, on a
subsequent restart of the broker we see this:
In this scenario I think it is reasonable to just emit a warning and have the
cleaner round up its first dirty offset to the base offset of the first
segment.
{code}
[kafka-server] [] [kafka-log-cleaner-thread-0], Error due to
java.lang.IllegalArgumentException: requirement failed: Last clean offset is
54770438 but segment base offset is 382844024 for log testtopic-0.
at scala.Predef$.require(Predef.scala:145)
at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:491)
at kafka.log.Cleaner.clean(LogCleaner.scala:288)
at
kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:202)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:187)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1070) Auto-assign node id
[
https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14073665#comment-14073665
]
Jon Bringhurst edited comment on KAFKA-1070 at 7/24/14 9:16 PM:
[~clarkhaskins], [~sriharsha], having ReservedBrokerIdMaxValue as a
configurable option (perhaps with a default of 1000) would probably solve
clark's use case.
-Also, if I'm reading this right, it will ignore the broker ID if it's set in
the config but out of the reserved range (set to -1). It will then create a
generated broker ID to overwrite the one in the config. It might be nice to
have the broker error out on startup if a broker.id is set in the config, but
is out of range. A new broker ID should only be generated if an id isn't
specified in the config AND the meta.properties doesn't exist (otherwise the
config file wouldn't match the actual broker ID).- Edit: nevermind, my Scala
reading skills aren't up to par. This isn't the case here.
was (Author: jonbringhurst):
[~clarkhaskins], [~sriharsha], having ReservedBrokerIdMaxValue as a
configurable option (perhaps with a default of 1000) would probably solve
clark's use case.
Also, if I'm reading this right, it will ignore the broker ID if it's set in
the config but out of the reserved range (set to -1). It will then create a
generated broker ID to overwrite the one in the config. It might be nice to
have the broker error out on startup if a broker.id is set in the config, but
is out of range. A new broker ID should only be generated if an id isn't
specified in the config AND the meta.properties doesn't exist (otherwise the
config file wouldn't match the actual broker ID).
{noformat}
var brokerId: Int = if (props.containsKey(broker.id))
props.getIntInRange(broker.id, (0, ReservedBrokerIdMaxValue)) else -1
... later on ...
} else if(metaBrokerIdSet.size == 0) {
if(brokerId 0) {
brokerId = ZkUtils.getBrokerSequenceId(zkClient)
} else {
{noformat}
Auto-assign node id
---
Key: KAFKA-1070
URL: https://issues.apache.org/jira/browse/KAFKA-1070
Project: Kafka
Issue Type: Bug
Reporter: Jay Kreps
Assignee: Sriharsha Chintalapani
Labels: usability
Attachments: KAFKA-1070.patch, KAFKA-1070_2014-07-19_16:06:13.patch,
KAFKA-1070_2014-07-22_11:34:18.patch
It would be nice to have Kafka brokers auto-assign node ids rather than
having that be a configuration. Having a configuration is irritating because
(1) you have to generate a custom config for each broker and (2) even though
it is in configuration, changing the node id can cause all kinds of bad
things to happen.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (KAFKA-1070) Auto-assign node id
[
https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14073665#comment-14073665
]
Jon Bringhurst commented on KAFKA-1070:
---
[~clarkhaskins], [~sriharsha], having ReservedBrokerIdMaxValue as a
configurable option (perhaps with a default of 1000) would probably solve
clark's use case.
Also, if I'm reading this right, it will ignore the broker ID if it's set in
the config but out of the reserved range (set to -1). It will then create a
generated broker ID to overwrite the one in the config. It might be nice to
have the broker error out on startup if a broker.id is set in the config, but
is out of range. A new broker ID should only be generated if an id isn't
specified in the config AND the meta.properties doesn't exist (otherwise the
config file wouldn't match the actual broker ID).
{noformat}
var brokerId: Int = if (props.containsKey(broker.id))
props.getIntInRange(broker.id, (0, ReservedBrokerIdMaxValue)) else -1
... later on ...
} else if(metaBrokerIdSet.size == 0) {
if(brokerId 0) {
brokerId = ZkUtils.getBrokerSequenceId(zkClient)
} else {
{noformat}
Auto-assign node id
---
Key: KAFKA-1070
URL: https://issues.apache.org/jira/browse/KAFKA-1070
Project: Kafka
Issue Type: Bug
Reporter: Jay Kreps
Assignee: Sriharsha Chintalapani
Labels: usability
Attachments: KAFKA-1070.patch, KAFKA-1070_2014-07-19_16:06:13.patch,
KAFKA-1070_2014-07-22_11:34:18.patch
It would be nice to have Kafka brokers auto-assign node ids rather than
having that be a configuration. Having a configuration is irritating because
(1) you have to generate a custom config for each broker and (2) even though
it is in configuration, changing the node id can cause all kinds of bad
things to happen.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (KAFKA-1478) Remove spam comment from wiki
Jon Bringhurst created KAFKA-1478: - Summary: Remove spam comment from wiki Key: KAFKA-1478 URL: https://issues.apache.org/jira/browse/KAFKA-1478 Project: Kafka Issue Type: Bug Reporter: Jon Bringhurst A spam comment exists on the wiki at: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication?focusedCommentId=33294292#comment-33294292 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1464) Add a throttling option to the Kafka replication tool
[ https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005052#comment-14005052 ] Jon Bringhurst commented on KAFKA-1464: --- Although this would be nice to have, something similar exists as part of linux. You can accomplish the same type of thing by first adding the process into a net_cls cgroup. Then, you can use the tc command to classify the marked packets into an htb qdisc (possibly with an stb further down the tree to completely prevent starvation) to throttle the packets coming from kafka. * https://www.kernel.org/doc/Documentation/cgroups/net_cls.txt * http://www.tldp.org/ * http://linux.die.net/man/8/tc The blkio cgroup works in a similar way to throttle disk io. Add a throttling option to the Kafka replication tool - Key: KAFKA-1464 URL: https://issues.apache.org/jira/browse/KAFKA-1464 Project: Kafka Issue Type: New Feature Components: replication Affects Versions: 0.8.0 Reporter: Marcos Juarez Assignee: Neha Narkhede Priority: Minor Labels: replication, replication-tools When performing replication on new nodes of a Kafka cluster, the replication process will use all available resources to replicate as fast as possible. This causes performance issues (mostly disk IO and sometimes network bandwidth) when doing this in a production environment, in which you're trying to serve downstream applications, at the same time you're performing maintenance on the Kafka cluster. An option to throttle the replication to a specific rate (in either MB/s or activities/second) would help production systems to better handle maintenance tasks while still serving downstream applications. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (KAFKA-1464) Add a throttling option to the Kafka replication tool
[ https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005052#comment-14005052 ] Jon Bringhurst edited comment on KAFKA-1464 at 5/21/14 6:33 PM: Although this would be nice to have, something similar exists as part of linux. You can accomplish the same type of thing by first adding the process into a net_cls cgroup. Then, you can use the tc command to classify the marked packets into an htb qdisc (possibly with an sfq further down the tree to completely prevent starvation) to throttle the packets coming from kafka. * https://www.kernel.org/doc/Documentation/cgroups/net_cls.txt * http://www.tldp.org/ * http://linux.die.net/man/8/tc The blkio cgroup works in a similar way to throttle disk io. was (Author: jonbringhurst): Although this would be nice to have, something similar exists as part of linux. You can accomplish the same type of thing by first adding the process into a net_cls cgroup. Then, you can use the tc command to classify the marked packets into an htb qdisc (possibly with an stb further down the tree to completely prevent starvation) to throttle the packets coming from kafka. * https://www.kernel.org/doc/Documentation/cgroups/net_cls.txt * http://www.tldp.org/ * http://linux.die.net/man/8/tc The blkio cgroup works in a similar way to throttle disk io. Add a throttling option to the Kafka replication tool - Key: KAFKA-1464 URL: https://issues.apache.org/jira/browse/KAFKA-1464 Project: Kafka Issue Type: New Feature Components: replication Affects Versions: 0.8.0 Reporter: Marcos Juarez Assignee: Neha Narkhede Priority: Minor Labels: replication, replication-tools When performing replication on new nodes of a Kafka cluster, the replication process will use all available resources to replicate as fast as possible. This causes performance issues (mostly disk IO and sometimes network bandwidth) when doing this in a production environment, in which you're trying to serve downstream applications, at the same time you're performing maintenance on the Kafka cluster. An option to throttle the replication to a specific rate (in either MB/s or activities/second) would help production systems to better handle maintenance tasks while still serving downstream applications. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (KAFKA-1464) Add a throttling option to the Kafka replication tool
[ https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005052#comment-14005052 ] Jon Bringhurst edited comment on KAFKA-1464 at 5/21/14 6:34 PM: Although this would be nice to have, something similar exists as part of linux. You can accomplish the same type of thing by first adding the process into a net_cls cgroup. Then, you can use the tc command to classify the marked packets into an htb qdisc (possibly with an sfq further down the tree to completely prevent starvation) to throttle the packets coming from kafka. * https://www.kernel.org/doc/Documentation/cgroups/net_cls.txt * http://www.tldp.org/ * http://linux.die.net/man/8/tc * http://lartc.org/manpages/tc-htb.html * http://lartc.org/manpages/tc-sfq.html The blkio cgroup works in a similar way to throttle disk io. was (Author: jonbringhurst): Although this would be nice to have, something similar exists as part of linux. You can accomplish the same type of thing by first adding the process into a net_cls cgroup. Then, you can use the tc command to classify the marked packets into an htb qdisc (possibly with an sfq further down the tree to completely prevent starvation) to throttle the packets coming from kafka. * https://www.kernel.org/doc/Documentation/cgroups/net_cls.txt * http://www.tldp.org/ * http://linux.die.net/man/8/tc The blkio cgroup works in a similar way to throttle disk io. Add a throttling option to the Kafka replication tool - Key: KAFKA-1464 URL: https://issues.apache.org/jira/browse/KAFKA-1464 Project: Kafka Issue Type: New Feature Components: replication Affects Versions: 0.8.0 Reporter: Marcos Juarez Assignee: Neha Narkhede Priority: Minor Labels: replication, replication-tools When performing replication on new nodes of a Kafka cluster, the replication process will use all available resources to replicate as fast as possible. This causes performance issues (mostly disk IO and sometimes network bandwidth) when doing this in a production environment, in which you're trying to serve downstream applications, at the same time you're performing maintenance on the Kafka cluster. An option to throttle the replication to a specific rate (in either MB/s or activities/second) would help production systems to better handle maintenance tasks while still serving downstream applications. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (KAFKA-1464) Add a throttling option to the Kafka replication tool
[ https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005052#comment-14005052 ] Jon Bringhurst edited comment on KAFKA-1464 at 5/21/14 6:35 PM: Although this would be nice to have, something similar exists as part of linux. You can accomplish the same type of thing by first adding the process into a net_cls cgroup. Then, you can use the tc command to classify the marked packets into an htb qdisc (possibly with an sfq further down the tree to completely prevent starvation) to throttle the packets coming from kafka. * https://www.kernel.org/doc/Documentation/cgroups/net_cls.txt * http://linux.die.net/man/8/tc * http://lartc.org/manpages/tc-htb.html * http://lartc.org/manpages/tc-sfq.html The blkio cgroup works in a similar way to throttle disk io. was (Author: jonbringhurst): Although this would be nice to have, something similar exists as part of linux. You can accomplish the same type of thing by first adding the process into a net_cls cgroup. Then, you can use the tc command to classify the marked packets into an htb qdisc (possibly with an sfq further down the tree to completely prevent starvation) to throttle the packets coming from kafka. * https://www.kernel.org/doc/Documentation/cgroups/net_cls.txt * http://www.tldp.org/ * http://linux.die.net/man/8/tc * http://lartc.org/manpages/tc-htb.html * http://lartc.org/manpages/tc-sfq.html The blkio cgroup works in a similar way to throttle disk io. Add a throttling option to the Kafka replication tool - Key: KAFKA-1464 URL: https://issues.apache.org/jira/browse/KAFKA-1464 Project: Kafka Issue Type: New Feature Components: replication Affects Versions: 0.8.0 Reporter: Marcos Juarez Assignee: Neha Narkhede Priority: Minor Labels: replication, replication-tools When performing replication on new nodes of a Kafka cluster, the replication process will use all available resources to replicate as fast as possible. This causes performance issues (mostly disk IO and sometimes network bandwidth) when doing this in a production environment, in which you're trying to serve downstream applications, at the same time you're performing maintenance on the Kafka cluster. An option to throttle the replication to a specific rate (in either MB/s or activities/second) would help production systems to better handle maintenance tasks while still serving downstream applications. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1428) FIS not closed after Properties.load
[ https://issues.apache.org/jira/browse/KAFKA-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Bringhurst updated KAFKA-1428: -- Attachment: KAFKA-1428.patch FIS not closed after Properties.load Key: KAFKA-1428 URL: https://issues.apache.org/jira/browse/KAFKA-1428 Project: Kafka Issue Type: Bug Reporter: Jon Bringhurst Priority: Minor Attachments: KAFKA-1428.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (KAFKA-1428) FIS not closed after Properties.load
Jon Bringhurst created KAFKA-1428: - Summary: FIS not closed after Properties.load Key: KAFKA-1428 URL: https://issues.apache.org/jira/browse/KAFKA-1428 Project: Kafka Issue Type: Bug Reporter: Jon Bringhurst Priority: Minor Attachments: KAFKA-1428.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1428) FIS not closed after Properties.load
[ https://issues.apache.org/jira/browse/KAFKA-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Bringhurst updated KAFKA-1428: -- Description: A FileInputStream is not being closed after using it for a Properties.load. FIS not closed after Properties.load Key: KAFKA-1428 URL: https://issues.apache.org/jira/browse/KAFKA-1428 Project: Kafka Issue Type: Bug Reporter: Jon Bringhurst Priority: Minor Attachments: KAFKA-1428.patch A FileInputStream is not being closed after using it for a Properties.load. -- This message was sent by Atlassian JIRA (v6.2#6252)
