[jira] [Updated] (KAFKA-2439) Add MirrorMakerService to ducktape system tests
[ https://issues.apache.org/jira/browse/KAFKA-2439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-2439: - Reviewer: Ewen Cheslack-Postava Add MirrorMakerService to ducktape system tests --- Key: KAFKA-2439 URL: https://issues.apache.org/jira/browse/KAFKA-2439 Project: Kafka Issue Type: Sub-task Components: system tests Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2439) Add MirrorMakerService to ducktape system tests
Geoffrey Anderson created KAFKA-2439: Summary: Add MirrorMakerService to ducktape system tests Key: KAFKA-2439 URL: https://issues.apache.org/jira/browse/KAFKA-2439 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2408) (new) system tests: ConsoleConsumerService occasionally fails to register consumed message
[ https://issues.apache.org/jira/browse/KAFKA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-2408: - Reviewer: Ewen Cheslack-Postava Description: There have been a few spurious failures in ReplicationTest.test_hard_bounce, where it was reported that a few of the acked messages were not consumed. Checking the logs, however, it is clear that they were consumed, but ConsoleConsumerService failed to parse. Lines causing parsing failure looks something like: 779725[2015-08-03 07:25:47,757] ERROR [ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1], Error for partition [test_topic,0] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) (i.e. the consumed message, and a log message appear on the same line) ConsoleConsumerService simply tries to strip each line of whitespace and parse as an integer, which will clearly fail in this case. Solution should either redirect stderr elsewhere or update parsing to handle this. was: There have been a few spurious failures in ReplicationTest.test_hard_bounce, where it was reported that a few of the acked messages were not consumed. Checking the logs, however, it is clear that they were consumed, but ConsoleConsumerService failed to parse. Lines causing parsing failure looks something like: 779725[2015-08-03 07:25:47,757] ERROR [ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1], Error for partition [test_topic,0] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) (i.e. the consumed message, and a log message appear on the same line) ConsoleConsumerService simply tries to strip each line of whitespace and parse as an integer, which will clearly fail in this case. Solution should either redirect stderr elsewhere or update parsing to handle this. (new) system tests: ConsoleConsumerService occasionally fails to register consumed message -- Key: KAFKA-2408 URL: https://issues.apache.org/jira/browse/KAFKA-2408 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 There have been a few spurious failures in ReplicationTest.test_hard_bounce, where it was reported that a few of the acked messages were not consumed. Checking the logs, however, it is clear that they were consumed, but ConsoleConsumerService failed to parse. Lines causing parsing failure looks something like: 779725[2015-08-03 07:25:47,757] ERROR [ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1], Error for partition [test_topic,0] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) (i.e. the consumed message, and a log message appear on the same line) ConsoleConsumerService simply tries to strip each line of whitespace and parse as an integer, which will clearly fail in this case. Solution should either redirect stderr elsewhere or update parsing to handle this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2408) (new) system tests: ConsoleConsumerService occasionally fails to register consumed message
Geoffrey Anderson created KAFKA-2408: Summary: (new) system tests: ConsoleConsumerService occasionally fails to register consumed message Key: KAFKA-2408 URL: https://issues.apache.org/jira/browse/KAFKA-2408 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 There have been a few spurious failures in ReplicationTest.test_hard_bounce, where it was reported that a few of the acked messages were not consumed. Checking the logs, however, it is clear that they were consumed, but ConsoleConsumerService failed to parse. Lines causing parsing failure looks something like: 779725[2015-08-03 07:25:47,757] ERROR [ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1], Error for partition [test_topic,0] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) (i.e. the consumed message, and a log message appear on the same line) ConsoleConsumerService simply tries to strip each line of whitespace and parse as an integer, which will clearly fail in this case. Solution should either redirect stderr elsewhere or update parsing to handle this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2330) Vagrantfile sets global configs instead of per-provider override configs
[ https://issues.apache.org/jira/browse/KAFKA-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629060#comment-14629060 ] Geoffrey Anderson commented on KAFKA-2330: -- Sorry, for reference, the aws machine I tested this on had: $ vagrant --version Vagrant 1.7.2 $ vagrant plugin list vagrant-aws (0.6.0) vagrant-hostmanager (1.5.0) vagrant-share (1.1.3, system) Vagrantfile sets global configs instead of per-provider override configs Key: KAFKA-2330 URL: https://issues.apache.org/jira/browse/KAFKA-2330 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Ewen Cheslack-Postava Assignee: Ewen Cheslack-Postava Priority: Minor Fix For: 0.8.3 Attachments: KAFKA-2330.patch There's a couple of minor incorrect usages of the global configuration object in the Vagrantfile inside provider-specific override blocks where we should be using the override config object. Two end up being harmless since they have no affect on other providers (but should still be corrected). The third results in using rsync when using Virtualbox, which is unnecessary, slower, and requires copying the entire kafka directory to every VM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2330) Vagrantfile sets global configs instead of per-provider override configs
[ https://issues.apache.org/jira/browse/KAFKA-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628928#comment-14628928 ] Geoffrey Anderson commented on KAFKA-2330: -- +1 from me I confirmed on my mac that changes in the kafka directory are immediately visible within the vm's /vagrant directory, and that aws requires rsync. Note that the rsync excludes don't work, but this may actually be a problem with vagrant-aws: https://github.com/mitchellh/vagrant-aws/issues/304 At first I thought this might be an issue with the variable name 'vagrant_excludes': the current Vagrant docs show 'vagrant__exclude' (double underscore, singular - http://docs.vagrantup.com/v2/synced-folders/rsync.html), however you can find both usages in a quick google search, so it's not clear to me which if any is incorrect. I did a quick trial of both variable names, and neither seems to work. Vagrantfile sets global configs instead of per-provider override configs Key: KAFKA-2330 URL: https://issues.apache.org/jira/browse/KAFKA-2330 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Ewen Cheslack-Postava Assignee: Ewen Cheslack-Postava Priority: Minor Fix For: 0.8.3 Attachments: KAFKA-2330.patch There's a couple of minor incorrect usages of the global configuration object in the Vagrantfile inside provider-specific override blocks where we should be using the override config object. Two end up being harmless since they have no affect on other providers (but should still be corrected). The third results in using rsync when using Virtualbox, which is unnecessary, slower, and requires copying the entire kafka directory to every VM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2327) broker doesn't start if config defines advertised.host but not advertised.port
[
https://issues.apache.org/jira/browse/KAFKA-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoffrey Anderson updated KAFKA-2327:
-
Description:
To reproduce locally, in server.properties, define advertised.host and
port, but not advertised.port
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.init(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
getString(KafkaConfig.AdvertisedHostNameProp) + : +
getInt(KafkaConfig.AdvertisedPortProp))
-
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
advertisedHostName + : + advertisedPort
was:
To reproduce locally, in server.properties, define advertised.host and
port, but not advertised.port
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.init(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
getString(KafkaConfig.AdvertisedHostNameProp) + : +
getInt(KafkaConfig.AdvertisedPortProp))
-
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
advertisedHostName + : + advertisedPort
broker doesn't start if config defines advertised.host but not advertised.port
--
Key: KAFKA-2327
URL:
[jira] [Created] (KAFKA-2327) broker doesn't start if config defines advertised.host but not advertised.port
Geoffrey Anderson created KAFKA-2327:
Summary: broker doesn't start if config defines advertised.host
but not advertised.port
Key: KAFKA-2327
URL: https://issues.apache.org/jira/browse/KAFKA-2327
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.3
Reporter: Geoffrey Anderson
Assignee: Geoffrey Anderson
Priority: Minor
To reproduce locally, in server.properties, define advertised.host and
port, but not advertised.port
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.init(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
getString(KafkaConfig.AdvertisedHostNameProp) + : +
getInt(KafkaConfig.AdvertisedPortProp))
-
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
advertisedHostName + : + advertisedPort
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2327) broker doesn't start if config defines advertised.host but not advertised.port
[
https://issues.apache.org/jira/browse/KAFKA-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoffrey Anderson updated KAFKA-2327:
-
Description:
To reproduce locally, in server.properties, define advertised.host and
port, but not advertised.port
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.init(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
{code}
...
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
getString(KafkaConfig.AdvertisedHostNameProp) + : +
getInt(KafkaConfig.AdvertisedPortProp))
...
{code}
-
{code}
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
advertisedHostName + : + advertisedPort
{code}
was:
To reproduce locally, in server.properties, define advertised.host and
port, but not advertised.port
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.init(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
getString(KafkaConfig.AdvertisedHostNameProp) + : +
getInt(KafkaConfig.AdvertisedPortProp))
-
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
advertisedHostName + : + advertisedPort
broker doesn't start if config defines advertised.host but not advertised.port
--
Key: KAFKA-2327
URL:
[jira] [Commented] (KAFKA-2327) broker doesn't start if config defines advertised.host but not advertised.port
[
https://issues.apache.org/jira/browse/KAFKA-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621091#comment-14621091
]
Geoffrey Anderson commented on KAFKA-2327:
--
Oh quick reply! Yup I just opened a PR for the patch :)
broker doesn't start if config defines advertised.host but not advertised.port
--
Key: KAFKA-2327
URL: https://issues.apache.org/jira/browse/KAFKA-2327
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.3
Reporter: Geoffrey Anderson
Assignee: Geoffrey Anderson
Priority: Minor
To reproduce locally, in server.properties, define advertised.host and
port, but not advertised.port
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.init(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
{code}
...
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
getString(KafkaConfig.AdvertisedHostNameProp) + : +
getInt(KafkaConfig.AdvertisedPortProp))
...
{code}
-
{code}
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
advertisedHostName + : + advertisedPort
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2276) Initial patch for KIP-25
[ https://issues.apache.org/jira/browse/KAFKA-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596624#comment-14596624 ] Geoffrey Anderson commented on KAFKA-2276: -- Pull request is here: https://github.com/apache/kafka/pull/70 Initial patch for KIP-25 Key: KAFKA-2276 URL: https://issues.apache.org/jira/browse/KAFKA-2276 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Submit initial patch for KIP-25 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements) This patch should contain a few Service classes and a few tests which can serve as examples -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-2276) Initial patch for KIP-25
[ https://issues.apache.org/jira/browse/KAFKA-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596624#comment-14596624 ] Geoffrey Anderson edited comment on KAFKA-2276 at 6/22/15 9:00 PM: --- Pull request is here: https://github.com/apache/kafka/pull/70 More info: https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests was (Author: granders): Pull request is here: https://github.com/apache/kafka/pull/70 Initial patch for KIP-25 Key: KAFKA-2276 URL: https://issues.apache.org/jira/browse/KAFKA-2276 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Submit initial patch for KIP-25 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements) This patch should contain a few Service classes and a few tests which can serve as examples -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2276) Initial patch for KIP-25
Geoffrey Anderson created KAFKA-2276: Summary: Initial patch for KIP-25 Key: KAFKA-2276 URL: https://issues.apache.org/jira/browse/KAFKA-2276 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Submit initial patch for KIP-25 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements) This patch should contain a few Service classes and a few tests which can serve as examples -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2259) port offset_management_testsuite
Geoffrey Anderson created KAFKA-2259: Summary: port offset_management_testsuite Key: KAFKA-2259 URL: https://issues.apache.org/jira/browse/KAFKA-2259 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 Port to run on ducktape -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2258) Port mirrormaker_testsuite
Geoffrey Anderson created KAFKA-2258: Summary: Port mirrormaker_testsuite Key: KAFKA-2258 URL: https://issues.apache.org/jira/browse/KAFKA-2258 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Port mirrormaker_testsuite to run on ducktape -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2257) port replication_testsuite
Geoffrey Anderson created KAFKA-2257: Summary: port replication_testsuite Key: KAFKA-2257 URL: https://issues.apache.org/jira/browse/KAFKA-2257 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 Port subset of replication_testsuite to run on ducktape. Details to follow -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2256) Port system tests
Geoffrey Anderson created KAFKA-2256: Summary: Port system tests Key: KAFKA-2256 URL: https://issues.apache.org/jira/browse/KAFKA-2256 Project: Kafka Issue Type: Improvement Components: system tests Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 This is a tracking issue for the system test suites to be ported per KIP-25 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1347) Create a system test for network partitions
[ https://issues.apache.org/jira/browse/KAFKA-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541125#comment-14541125 ] Geoffrey Anderson commented on KAFKA-1347: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake Create a system test for network partitions --- Key: KAFKA-1347 URL: https://issues.apache.org/jira/browse/KAFKA-1347 Project: Kafka Issue Type: Test Reporter: Jay Kreps We got some free and rather public QA here: http://aphyr.com/posts/293-call-me-maybe-kafka We have since added a configuration to disable unclean leader election which allows you to prefer consistency over availability when all brokers fail. This has some unit tests, but ultimately there is no reason to believe this works unless we have a fairly aggressive system test case for it. https://cwiki.apache.org/confluence/display/KAFKA/Kafka+System+Tests It would be good to add support for network partitions. I don't think we actually need to try to use the jepsen stuff directly, we can just us the underlying tools it uses--iptables and tc. These are linux specific, but that is prolly okay. You can see these at work here: https://github.com/aphyr/jepsen/blob/master/src/jepsen/control/net.clj Having this would help provide better evidence that this works now, and would keep it working in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1604) System Test for Transaction Management
[ https://issues.apache.org/jira/browse/KAFKA-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541123#comment-14541123 ] Geoffrey Anderson commented on KAFKA-1604: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake System Test for Transaction Management -- Key: KAFKA-1604 URL: https://issues.apache.org/jira/browse/KAFKA-1604 Project: Kafka Issue Type: Test Reporter: Dong Lin Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1604_2014-08-19_17:31:16.patch, KAFKA-1604_2014-08-19_21:07:35.patch Perform end-to-end transaction management test in the following steps: 1) Start Zookeeper. 2) Start multiple brokers. 3) Create topic. 4) Start transaction-aware ProducerPerformance to generate transactional messages to topic. 5) Start transaction-aware ConsoleConsumer to read messages from topic. 6) Bounce brokers (optional). 7) Verify that same number of messages are sent and received. This patch depends on KAFKA-1524, KAFKA-1526 and KAFKA-1601. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2003) Add upgrade tests
[ https://issues.apache.org/jira/browse/KAFKA-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541121#comment-14541121 ] Geoffrey Anderson commented on KAFKA-2003: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake Add upgrade tests - Key: KAFKA-2003 URL: https://issues.apache.org/jira/browse/KAFKA-2003 Project: Kafka Issue Type: Improvement Components: system tests Reporter: Gwen Shapira Assignee: Ashish K Singh To test protocol changes, compatibility and upgrade process, we need a good way to test different versions of the product together and to test end-to-end upgrade process. For example, for 0.8.2 to 0.8.3 test we want to check: * Can we start a cluster with a mix of 0.8.2 and 0.8.3 brokers? * Can a cluster of 0.8.3 brokers bump the protocol level one broker at a time? * Can 0.8.2 clients run against a cluster of 0.8.3 brokers? There are probably more questions. But an automated framework that can test those and report results will be a good start. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1918) System test for ZooKeeper quorum failure scenarios
[
https://issues.apache.org/jira/browse/KAFKA-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541127#comment-14541127
]
Geoffrey Anderson commented on KAFKA-1918:
--
Just making sure you're aware of work we're doing at Confluent on system tests.
I'll be posting a KIP for this soon, but here's some info:
The original plan is sketched here:
https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements
This is the core library/test framework (WIP) which aids in writing and running
the tests
https://github.com/confluentinc/ducktape/
This has system tests we've written to date for the Confluent Platform
https://github.com/confluentinc/muckrake
System test for ZooKeeper quorum failure scenarios
--
Key: KAFKA-1918
URL: https://issues.apache.org/jira/browse/KAFKA-1918
Project: Kafka
Issue Type: Test
Reporter: Omid Aladini
Following up on the [conversation on the mailing
list|http://mail-archives.apache.org/mod_mbox/kafka-users/201502.mbox/%3CCAHwHRrX3SAWDUGF5LjU4rrMUsqv%3DtJcyjX7OENeL5C_V5o3tCw%40mail.gmail.com%3E],
the FAQ writes:
{quote}
Once the Zookeeper quorum is down, brokers could result in a bad state and
could not normally serve client requests, etc. Although when Zookeeper quorum
recovers, the Kafka brokers should be able to resume to normal state
automatically, _there are still a few +corner cases+ the they cannot and a
hard kill-and-recovery is required to bring it back to normal_. Hence it is
recommended to closely monitor your zookeeper cluster and provision it so
that it is performant.
{quote}
As ZK quorum failures are inevitable (due to rolling upgrades of ZK, leader
hardware failure, etc), it would be great to identify the corner cases (if
they still exist) and fix them if necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-976) Order-Preserving Mirror Maker Testcase
[ https://issues.apache.org/jira/browse/KAFKA-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541124#comment-14541124 ] Geoffrey Anderson commented on KAFKA-976: - Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake Order-Preserving Mirror Maker Testcase -- Key: KAFKA-976 URL: https://issues.apache.org/jira/browse/KAFKA-976 Project: Kafka Issue Type: Test Reporter: Guozhang Wang Assignee: John Fung Attachments: kafka-976-v1.patch A new testcase (5007) for mirror_maker_testsuite is needed for the key-dependent order-preserving mirror maker, this is related to KAFKA-957. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1904) run sanity failed test
[ https://issues.apache.org/jira/browse/KAFKA-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541120#comment-14541120 ] Geoffrey Anderson commented on KAFKA-1904: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake run sanity failed test -- Key: KAFKA-1904 URL: https://issues.apache.org/jira/browse/KAFKA-1904 Project: Kafka Issue Type: Bug Components: system tests Reporter: Joe Stein Priority: Blocker Fix For: 0.8.3 Attachments: run_sanity.log.gz _test_case_name : testcase_1 _test_class_name : ReplicaBasicTest arg : bounce_broker : true arg : broker_type : leader arg : message_producing_free_time_sec : 15 arg : num_iteration : 2 arg : num_messages_to_produce_per_producer_call : 50 arg : num_partition : 2 arg : replica_factor : 3 arg : sleep_seconds_between_producer_calls : 1 validation_status : Test completed : FAILED -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1898) compatibility testing framework
[
https://issues.apache.org/jira/browse/KAFKA-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541122#comment-14541122
]
Geoffrey Anderson commented on KAFKA-1898:
--
Just making sure you're aware of work we're doing at Confluent on system tests.
I'll be posting a KIP for this soon, but here's some info:
The original plan is sketched here:
https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements
This is the core library/test framework (WIP) which aids in writing and running
the tests
https://github.com/confluentinc/ducktape/
This has system tests we've written to date for the Confluent Platform
https://github.com/confluentinc/muckrake
compatibility testing framework
Key: KAFKA-1898
URL: https://issues.apache.org/jira/browse/KAFKA-1898
Project: Kafka
Issue Type: Bug
Reporter: Joe Stein
Fix For: 0.8.3
Attachments: cctk.png
There are a few different scenarios where you want/need to know the
status/state of a client library that works with Kafka. Client library
development is not just about supporting the wire protocol but also the
implementations around specific interactions of the API. The API has
blossomed into a robust set of producer, consumer, broker and administrative
calls all of which have layers of logic above them. A Client Library may
choose to deviate from the path the project sets out and that is ok. The goal
of this ticket is to have a system for Kafka that can help to explain what
the library is or isn't doing (regardless of what it claims).
The idea behind this stems in being able to quickly/easily/succinctly analyze
the topic message data. Once you can analyze the topic(s) message you can
gather lots of information about what the client library is doing, is not
doing and such. There are a few components to this.
1) dataset-generator
Test Kafka dataset generation tool. Generates a random text file with given
params:
--filename, -f - output file name.
--filesize, -s - desired size of output file. The actual size will always be
a bit larger (with a maximum size of $filesize + $max.length - 1)
--min.length, -l - minimum generated entry length.
--max.length, -h - maximum generated entry length.
Usage:
./gradlew build
java -jar dataset-generator/build/libs/dataset-generator-*.jar -s 10 -l 2
-h 20
2) dataset-producer
Test Kafka dataset producer tool. Able to produce the given dataset to Kafka
or Syslog server. The idea here is you already have lots of data sets that
you want to test different things for. You might have different sized
messages, formats, etc and want a repeatable benchmark to run and re-run the
testing on. You could just have a days worth of data and just choose to
replay it. The CCTK idea is that you are always starting from CONSUME in
your state of library. If your library is only producing then you will fail a
bunch of tests and that might be ok for people.
Accepts following params:
{code}
--filename, -f - input file name.
--kafka, -k - Kafka broker address in host:port format. If this parameter is
set, --producer.config and --topic must be set too (otherwise they're
ignored).
--producer.config, -p - Kafka producer properties file location.
--topic, -t - Kafka topic to produce to.
--syslog, -s - Syslog server address. Format: protocol://host:port
(tcp://0.0.0.0:5140 or udp://0.0.0.0:5141 for example)
--loop, -l - flag to loop through file until shut off manually. False by
default.
Usage:
./gradlew build
java -jar dataset-producer/build/libs/dataset-producer-*.jar --filename
dataset --syslog tcp://0.0.0.0:5140 --loop true
{code}
3) extract
This step is good so you can save data and compare tests. It could also be
removed if folks are just looking for a real live test (and we could support
that too). Here we are taking data out of Kafka and putting it into
Cassandra (but other data stores can be used too and we should come up with a
way to abstract this out completely so folks could implement whatever they
wanted.
{code}
package ly.stealth.shaihulud.reader
import java.util.UUID
import com.datastax.spark.connector._
import com.datastax.spark.connector.cql.CassandraConnector
import consumer.kafka.MessageAndMetadata
import consumer.kafka.client.KafkaReceiver
import org.apache.spark._
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming._
import org.apache.spark.streaming.dstream.DStream
object Main extends App with Logging {
val parser = new scopt.OptionParser[ReaderConfiguration](spark-reader) {
head(Spark Reader for Kafka client applications, 1.0)
opt[String](testId) unbounded() optional() action { (x, c) =
c.copy(testId = x)
} text (Source topic with initial set of data)
[jira] [Commented] (KAFKA-1888) Add a rolling upgrade system test
[ https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541118#comment-14541118 ] Geoffrey Anderson commented on KAFKA-1888: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake Add a rolling upgrade system test --- Key: KAFKA-1888 URL: https://issues.apache.org/jira/browse/KAFKA-1888 Project: Kafka Issue Type: Improvement Components: system tests Reporter: Gwen Shapira Assignee: Abhishek Nigam Fix For: 0.9.0 Attachments: KAFKA-1888_2015-03-23_11:54:25.patch To help test upgrades and compatibility between versions, it will be cool to add a rolling-upgrade test to system tests: Given two versions (just a path to the jars?), check that you can do a rolling upgrade of the brokers from one version to another (using clients from the old version) without losing data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2171) System Test for Quotas
[ https://issues.apache.org/jira/browse/KAFKA-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541116#comment-14541116 ] Geoffrey Anderson commented on KAFKA-2171: -- Just pinging this thread to make sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake System Test for Quotas -- Key: KAFKA-2171 URL: https://issues.apache.org/jira/browse/KAFKA-2171 Project: Kafka Issue Type: Sub-task Reporter: Dong Lin Assignee: Dong Lin Original Estimate: 336h Remaining Estimate: 336h Motivation and goal: We want to make sure that following features are working properly for both consumer and producer: default quota, client-specific quota, and quota sharing per clientId. The tests and configuration described aims to cover most of the scenarios. More test cases with varying configurations (e.g. ackNum) can be added if there is good reason to do so. Initial setup and configuration: In all scenarios, we first create kafka brokers and topic as follows: - create two kafka broker processes (by default local) - create a topic with replication factor = 2 and ackNum = -1 - let max_read = max_write = 5MB. The test machine is expected to provide read (write) throughput at least max_read (max_write). - we consider two rates are approximately the same if they differ by at most 5%. Scenario 1: Validate that max_read and max_write are provided by the test machine(s) using 1 producer and 1 consumer 1) produce data to the topic without rate limit for 30 seconds 2) record the rate of producer 3) then, consume data from the topic without rate limit until he finishes 4) record the rate of consumer 5) verify that the data consumed is identical to the data produced 6) verify that producer rate = max_write and consumer rate = max_read Scenario 2: validate the effectiveness of default write and read quota using 1 producer and 1 consumer 1) configure brokers to use max_write/2 as default write quota and max_read/2 as default read quota 2) produce data to the topic for 30 seconds 3) record the rate of producer 4) then, consume data from the topic until he finishes 5) record the rate of consumer 6) verify that the data consumed is identical to the data produced 7) verify that recorded write (read) rate is within 5% of max_write/2 (max_read/2). Scenario 3: validate the effectiveness of client-specific write and read quota using 2 producers and 2 consumers 1) configure brokers to use max_write/2 as default write quota and max_read/2 as default read quota; configure brokers to use max_write/4 for producer_2 and max_read/4 for consumer_2 2) both producers produce data to the topic for 30 seconds. They use different clientId. 3) record the rate of producer 4) both consumers consume data from the topic until they finish. They use different clientId and groupId. 5) record the rate of consumer 6) verify that the data consumed is identical to the data produced 7) verify that producer_1 and producer_2 rates are approximately max_write/2 and max_write/4; verify that consumer_1 and consumer_2 rates are approximately max_read/2 and max_read/4. Scenario 4: validate the effectiveness of write and read quota sharing among clients of same clientId using 2 producers and 2 consumers. 1) configure brokers to use max_write/2 as default write quota and max_read/2 as default read quota 2) both producers produce data to the topic for 30 seconds. They use same clientId. 3) record the rate of producer 4) both consumers consume data from the topic until they finish. They use same clientId but different groupId. 5) record the rate of consumer 6) verify that the data consumed is identical to the data produced 7) verify that total rate of producer_1 and producer_2 is approximately max_write/2; verify that total rate of consumer_1 and consumer_2 is approximately max_read/2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1313) Support adding replicas to existing topic partitions
[
https://issues.apache.org/jira/browse/KAFKA-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347795#comment-14347795
]
Geoffrey Anderson commented on KAFKA-1313:
--
Quoting the Kafka docs:
The first step is to hand craft the custom reassignment plan in a json file-
cat increase-replication-factor.json
{version:1,
partitions:[{topic:foo,partition:0,replicas:[5,6,7]}]}
Then, use the json file with the --execute option to start the reassignment
process-
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181
--reassignment-json-file increase-replication-factor.json --execute
---
As I understand it, the pain point is hand crafting the reassignment plan.
From there on out, updating replication factor is functionally identical to any
other reassignment. So my proposal is to add another file type
(--replication-factor-json-file) to kafka-reassign-partitions.sh --generate
(and/or --rebalance per KIP 6) which allows users to specify desired
replication_factor per topic/partition.
For example:
cat update-replication-factor.json
{version:1,
partitions:[{topic:foo,partition:0,replication_factor:3}]}
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181
--replication-factor-json-file update-replication-factor.json --generate
The user would then feed generated output into kafka-reassign-partitions.sh
--execute as before.
[~nehanarkhede], let me know if this seems reasonable/is in the ballpark of
what you had in mind.
Support adding replicas to existing topic partitions
Key: KAFKA-1313
URL: https://issues.apache.org/jira/browse/KAFKA-1313
Project: Kafka
Issue Type: New Feature
Components: tools
Affects Versions: 0.8.0
Reporter: Marc Labbe
Assignee: Geoffrey Anderson
Priority: Critical
Labels: newbie++
Fix For: 0.9.0
There is currently no easy way to add replicas to an existing topic
partitions.
For example, topic create-test has been created with ReplicationFactor=1:
Topic:create-test PartitionCount:3ReplicationFactor:1 Configs:
Topic: create-test Partition: 0Leader: 1 Replicas: 1 Isr: 1
Topic: create-test Partition: 1Leader: 2 Replicas: 2 Isr: 2
Topic: create-test Partition: 2Leader: 3 Replicas: 3 Isr: 3
I would like to increase the ReplicationFactor=2 (or more) so it shows up
like this instead.
Topic:create-test PartitionCount:3ReplicationFactor:2 Configs:
Topic: create-test Partition: 0Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: create-test Partition: 1Leader: 2 Replicas: 2,3 Isr: 2,3
Topic: create-test Partition: 2Leader: 3 Replicas: 3,1 Isr: 3,1
Use cases for this:
- adding brokers and thus increase fault tolerance
- fixing human errors for topics created with wrong values
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346083#comment-14346083 ] Geoffrey Anderson commented on KAFKA-902: - Thanks for the feedback. Is there any reason to keep 0 - 0 behavior? In other words, is there any harm to making 0 less magical and perturbing it just like any other value? Also, I like the idea of scaling the upper bound on jitter with backoff_ms, though it seems like we still want some amount of jitter even if backoff_ms is small (in your example, if backoff_ms 5, then jitter is always 0) In that case, we might want to make sure that the jitter can be non-zero with something like max(3, min(20, 0.2 * backoff_ms)) But then we end up with a couple more semi-arbitrary parameters - a scaling parameter and a hard minimum. Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346149#comment-14346149 ] Geoffrey Anderson commented on KAFKA-902: - Created reviewboard https://reviews.apache.org/r/31715/diff/ against branch origin/trunk Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch, KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-902: Attachment: KAFKA-902.patch Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch, KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343607#comment-14343607 ] Geoffrey Anderson commented on KAFKA-902: - Created reviewboard https://reviews.apache.org/r/31633/diff/ against branch origin/trunk Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-902: Status: Patch Available (was: Open) Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-902: Attachment: KAFKA-902.patch Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-1700) examples directory - README and shell scripts are out of date
Geoffrey Anderson created KAFKA-1700: Summary: examples directory - README and shell scripts are out of date Key: KAFKA-1700 URL: https://issues.apache.org/jira/browse/KAFKA-1700 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Geoffrey Anderson Priority: Minor Fix For: 0.8.2 sbt build files were removed during resolution of KAFKA-1254, so the README under the examples directory should no longer make reference to sbt. Also, the paths added to CLASSPATH variable in the example shell script are no longer correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1700) examples directory - README and shell scripts are out of date
[ https://issues.apache.org/jira/browse/KAFKA-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-1700: - Attachment: KAFKA-1700.patch examples directory - README and shell scripts are out of date - Key: KAFKA-1700 URL: https://issues.apache.org/jira/browse/KAFKA-1700 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Geoffrey Anderson Priority: Minor Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1700.patch sbt build files were removed during resolution of KAFKA-1254, so the README under the examples directory should no longer make reference to sbt. Also, the paths added to CLASSPATH variable in the example shell script are no longer correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1700) examples directory - README and shell scripts are out of date
[ https://issues.apache.org/jira/browse/KAFKA-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167501#comment-14167501 ] Geoffrey Anderson commented on KAFKA-1700: -- Created reviewboard https://reviews.apache.org/r/26575/diff/ against branch origin/trunk examples directory - README and shell scripts are out of date - Key: KAFKA-1700 URL: https://issues.apache.org/jira/browse/KAFKA-1700 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Geoffrey Anderson Priority: Minor Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1700.patch sbt build files were removed during resolution of KAFKA-1254, so the README under the examples directory should no longer make reference to sbt. Also, the paths added to CLASSPATH variable in the example shell script are no longer correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
