[jira] [Commented] (KAFKA-3216) "Modifying topics" section incorrectly says you can't change replication factor.
[
https://issues.apache.org/jira/browse/KAFKA-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136152#comment-15136152
]
ASF GitHub Bot commented on KAFKA-3216:
---
GitHub user wushujames opened a pull request:
https://github.com/apache/kafka/pull/881
KAFKA-3216: "Modifying topics" section incorrectly says you can't change
replication factor.
Correct the text that said that you can't change the replication factor of
a topic.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wushujames/kafka KAFKA-3216
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/881.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #881
commit f6b2e73a0295bff0cb8a9108b4b3a1db29077526
Author: James Cheng
Date: 2016-02-07T05:40:30Z
Correct the text that said that you can't change the replication factor of
a topic.
> "Modifying topics" section incorrectly says you can't change replication
> factor.
>
>
> Key: KAFKA-3216
> URL: https://issues.apache.org/jira/browse/KAFKA-3216
> Project: Kafka
> Issue Type: Bug
>Reporter: James Cheng
>Assignee: James Cheng
>
> The "Modifying Topics" section of the docs
> (http://kafka.apache.org/documentation.html#basic_ops_modify_topic) says
> {quote} Kafka does not currently support reducing the number of partitions
> for a topic or changing the replication factor. {quote}
> But you *can* modify the replication factor. That second half of the sentence
> should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] kafka pull request: KAFKA-3216: "Modifying topics" section incorre...
GitHub user wushujames opened a pull request: https://github.com/apache/kafka/pull/881 KAFKA-3216: "Modifying topics" section incorrectly says you can't change replication factor. Correct the text that said that you can't change the replication factor of a topic. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wushujames/kafka KAFKA-3216 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/881.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #881 commit f6b2e73a0295bff0cb8a9108b4b3a1db29077526 Author: James Cheng Date: 2016-02-07T05:40:30Z Correct the text that said that you can't change the replication factor of a topic. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (KAFKA-2818) Clean up Controller Object on forced Resignation
[
https://issues.apache.org/jira/browse/KAFKA-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flavio Junqueira updated KAFKA-2818:
Assignee: Flavio Junqueira (was: Neha Narkhede)
Status: Open (was: Patch Available)
> Clean up Controller Object on forced Resignation
>
>
> Key: KAFKA-2818
> URL: https://issues.apache.org/jira/browse/KAFKA-2818
> Project: Kafka
> Issue Type: Bug
> Components: controller
>Affects Versions: 0.9.0.0
>Reporter: Matthew Bruce
>Assignee: Flavio Junqueira
>Priority: Minor
> Attachments: KAFKA-2818.patch
>
>
> Currently if the controller does a forced resignation (if an exception is
> caught during updateLeaderEpochAndSendRequest, SendUpdateMetadataRequest or
> shutdownBroker), the Zookeeper resignation callback function
> OnControllerResignation doesn't get a chance to execute which leaves some
> artifacts laying around. In particular the Sensors dont get cleaned up and
> if the Kafka broker ever gets re-elected as Controller it will fail due to
> some metrics already existing. An Error and stack trace of such an event is
> below.
> A forced resignation situation can be induced with a mis-config in
> broker.properties fairly easily, by settig only SASL_PLAINTTEXT listeners and
> setting inter.broker.protocol.version=0.8.2.X
> {code}
> listeners=SASL_PLAINTEXT://:9092
> inter.broker.protocol.version=0.8.2.X
> security.inter.broker.protocol=SASL_PLAINTEXT
> {code}
> {code}
> [2015-11-09 16:33:47,510] ERROR Error while electing or becoming leader on
> broker 182050300 (kafka.server.ZookeeperLeaderElector)
> java.lang.IllegalArgumentException: A metric named 'MetricName
> [name=connection-close-rate, group=controller-channel-metrics,
> description=Connections closed per second in the window.,
> tags={broker-id=182050300}]' already exists, can't register another one.
> at
> org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:285)
> at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:177)
> at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:162)
> at
> org.apache.kafka.common.network.Selector$SelectorMetrics.(Selector.java:578)
> at org.apache.kafka.common.network.Selector.(Selector.java:112)
> at
> kafka.controller.ControllerChannelManager.kafka$controller$ControllerChannelManager$$addNewBroker(ControllerChannelManager.scala:91)
> at
> kafka.controller.ControllerChannelManager$$anonfun$1.apply(ControllerChannelManager.scala:43)
> at
> kafka.controller.ControllerChannelManager$$anonfun$1.apply(ControllerChannelManager.scala:43)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
> at
> kafka.controller.ControllerChannelManager.(ControllerChannelManager.scala:43)
> at
> kafka.controller.KafkaController.startChannelManager(KafkaController.scala:819)
> at
> kafka.controller.KafkaController.initializeControllerContext(KafkaController.scala:747)
> at
> kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:330)
> at
> kafka.controller.KafkaController$$anonfun$1.apply$mcV$sp(KafkaController.scala:163)
> at
> kafka.server.ZookeeperLeaderElector.elect(ZookeeperLeaderElector.scala:84)
> at
> kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply$mcZ$sp(ZookeeperLeaderElector.scala:146)
> at
> kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:141)
> at
> kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:141)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
> at
> kafka.server.ZookeeperLeaderElector$LeaderChangeListener.handleDataDeleted(ZookeeperLeaderElector.scala:141)
> at org.I0Itec.zkclient.ZkClient$9.run(ZkClient.java:823)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2818) Clean up Controller Object on forced Resignation
[
https://issues.apache.org/jira/browse/KAFKA-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136147#comment-15136147
]
Flavio Junqueira commented on KAFKA-2818:
-
[~mbr...@blackberry.com] You're right that {{onControllerResignation()}} isn't
being called, but the problem is that
{{ZooKeeperLeaderElector.handleDataDeleted()}} should be invoking it via
{{onResigningAsLeader()}} and it isn't because {{amILeader}} is evaluating to
false. It is evaluating to false because the call to {{resign()}} is setting
{{leaderId}} to -1. I'm thinking that we shouldn't set {{leaderId}} to -1 in
the {{resign()}} call. {{leaderId}} to will be set to -1 if {{getControllerID}}
returns -1 when called inside {{elect}}.
What do you think?
> Clean up Controller Object on forced Resignation
>
>
> Key: KAFKA-2818
> URL: https://issues.apache.org/jira/browse/KAFKA-2818
> Project: Kafka
> Issue Type: Bug
> Components: controller
>Affects Versions: 0.9.0.0
>Reporter: Matthew Bruce
>Assignee: Neha Narkhede
>Priority: Minor
> Attachments: KAFKA-2818.patch
>
>
> Currently if the controller does a forced resignation (if an exception is
> caught during updateLeaderEpochAndSendRequest, SendUpdateMetadataRequest or
> shutdownBroker), the Zookeeper resignation callback function
> OnControllerResignation doesn't get a chance to execute which leaves some
> artifacts laying around. In particular the Sensors dont get cleaned up and
> if the Kafka broker ever gets re-elected as Controller it will fail due to
> some metrics already existing. An Error and stack trace of such an event is
> below.
> A forced resignation situation can be induced with a mis-config in
> broker.properties fairly easily, by settig only SASL_PLAINTTEXT listeners and
> setting inter.broker.protocol.version=0.8.2.X
> {code}
> listeners=SASL_PLAINTEXT://:9092
> inter.broker.protocol.version=0.8.2.X
> security.inter.broker.protocol=SASL_PLAINTEXT
> {code}
> {code}
> [2015-11-09 16:33:47,510] ERROR Error while electing or becoming leader on
> broker 182050300 (kafka.server.ZookeeperLeaderElector)
> java.lang.IllegalArgumentException: A metric named 'MetricName
> [name=connection-close-rate, group=controller-channel-metrics,
> description=Connections closed per second in the window.,
> tags={broker-id=182050300}]' already exists, can't register another one.
> at
> org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:285)
> at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:177)
> at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:162)
> at
> org.apache.kafka.common.network.Selector$SelectorMetrics.(Selector.java:578)
> at org.apache.kafka.common.network.Selector.(Selector.java:112)
> at
> kafka.controller.ControllerChannelManager.kafka$controller$ControllerChannelManager$$addNewBroker(ControllerChannelManager.scala:91)
> at
> kafka.controller.ControllerChannelManager$$anonfun$1.apply(ControllerChannelManager.scala:43)
> at
> kafka.controller.ControllerChannelManager$$anonfun$1.apply(ControllerChannelManager.scala:43)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
> at
> kafka.controller.ControllerChannelManager.(ControllerChannelManager.scala:43)
> at
> kafka.controller.KafkaController.startChannelManager(KafkaController.scala:819)
> at
> kafka.controller.KafkaController.initializeControllerContext(KafkaController.scala:747)
> at
> kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:330)
> at
> kafka.controller.KafkaController$$anonfun$1.apply$mcV$sp(KafkaController.scala:163)
> at
> kafka.server.ZookeeperLeaderElector.elect(ZookeeperLeaderElector.scala:84)
> at
> kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply$mcZ$sp(ZookeeperLeaderElector.scala:146)
> at
> kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:141)
> at
> kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:141)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
> at
> kafka.server.ZookeeperLeaderElector$LeaderChangeListener.handleDataDeleted(ZookeeperLeaderElector.scala:141)
> at org.I0Itec.zkclient.ZkClient$9.run(ZkClient.java:823)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KAFKA-3216) "Modifying topics" section incorrectly says you can't change replication factor.
James Cheng created KAFKA-3216:
--
Summary: "Modifying topics" section incorrectly says you can't
change replication factor.
Key: KAFKA-3216
URL: https://issues.apache.org/jira/browse/KAFKA-3216
Project: Kafka
Issue Type: Bug
Reporter: James Cheng
Assignee: James Cheng
The "Modifying Topics" section of the docs
(http://kafka.apache.org/documentation.html#basic_ops_modify_topic) says
{quote} Kafka does not currently support reducing the number of partitions for
a topic or changing the replication factor. {quote}
But you *can* modify the replication factor. That second half of the sentence
should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions
[
https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136071#comment-15136071
]
Jason Gustafson commented on KAFKA-3159:
Either should work, but perhaps it would be most useful at the moment to try
against 0.9.0.
> Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain
> conditions
> --
>
> Key: KAFKA-3159
> URL: https://issues.apache.org/jira/browse/KAFKA-3159
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 0.9.0.0
> Environment: Linux, Oracle JVM 8.
>Reporter: Rajiv Kurian
>Assignee: Jason Gustafson
> Fix For: 0.9.0.1
>
> Attachments: Memory-profile-patched-client.png, Screen Shot
> 2016-02-01 at 11.09.32 AM.png
>
>
> We are using the new kafka consumer with the following config (as logged by
> kafka)
> metric.reporters = []
> metadata.max.age.ms = 30
> value.deserializer = class
> org.apache.kafka.common.serialization.ByteArrayDeserializer
> group.id = myGroup.id
> partition.assignment.strategy =
> [org.apache.kafka.clients.consumer.RangeAssignor]
> reconnect.backoff.ms = 50
> sasl.kerberos.ticket.renew.window.factor = 0.8
> max.partition.fetch.bytes = 2097152
> bootstrap.servers = [myBrokerList]
> retry.backoff.ms = 100
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> ssl.keystore.type = JKS
> ssl.trustmanager.algorithm = PKIX
> enable.auto.commit = false
> ssl.key.password = null
> fetch.max.wait.ms = 1000
> sasl.kerberos.min.time.before.relogin = 6
> connections.max.idle.ms = 54
> ssl.truststore.password = null
> session.timeout.ms = 3
> metrics.num.samples = 2
> client.id =
> ssl.endpoint.identification.algorithm = null
> key.deserializer = class sf.kafka.VoidDeserializer
> ssl.protocol = TLS
> check.crcs = true
> request.timeout.ms = 4
> ssl.provider = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> ssl.keystore.location = null
> heartbeat.interval.ms = 3000
> auto.commit.interval.ms = 5000
> receive.buffer.bytes = 32768
> ssl.cipher.suites = null
> ssl.truststore.type = JKS
> security.protocol = PLAINTEXT
> ssl.truststore.location = null
> ssl.keystore.password = null
> ssl.keymanager.algorithm = SunX509
> metrics.sample.window.ms = 3
> fetch.min.bytes = 512
> send.buffer.bytes = 131072
> auto.offset.reset = earliest
> We use the consumer.assign() feature to assign a list of partitions and call
> poll in a loop. We have the following setup:
> 1. The messages have no key and we use the byte array deserializer to get
> byte arrays from the config.
> 2. The messages themselves are on an average about 75 bytes. We get this
> number by dividing the Kafka broker bytes-in metric by the messages-in metric.
> 3. Each consumer is assigned about 64 partitions of the same topic spread
> across three brokers.
> 4. We get very few messages per second maybe around 1-2 messages across all
> partitions on a client right now.
> 5. We have no compression on the topic.
> Our run loop looks something like this
> while (isRunning()) {
> ConsumerRecords records = null;
> try {
> // Here timeout is about 10 seconds, so it is pretty big.
> records = consumer.poll(timeout);
> } catch (Exception e) {
>// This never hits for us
> logger.error("Exception polling Kafka ", e);
> records = null;
> }
> if (records != null) {
> for (ConsumerRecord record : records) {
>// The handler puts the byte array on a very fast ring buffer
> so it barely takes any time.
> handler.handleMessage(ByteBuffer.wrap(record.value()));
> }
> }
> }
> With this setup our performance has taken a horrendous hit as soon as we
> started this one thread that just polls Kafka in a loop.
> I profiled the application using Java Mission Control and have a few insights.
> 1. There doesn't seem to be a single hotspot. The consumer just ends up using
> a lot of CPU for handing such a low number of messages. Our process was using
> 16% CPU before we added a single consumer and it went to 25% and above after.
> That's an increase of over 50% from a single consumer getting a single digit
> number of small messages per second. Here is an attachment of the cpu usage
> b
[jira] [Updated] (KAFKA-3190) KafkaProducer should not invoke callback in send()
[ https://issues.apache.org/jira/browse/KAFKA-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-3190: Status: Patch Available (was: Open) > KafkaProducer should not invoke callback in send() > -- > > Key: KAFKA-3190 > URL: https://issues.apache.org/jira/browse/KAFKA-3190 > Project: Kafka > Issue Type: Bug > Components: clients, producer >Affects Versions: 0.9.0.0 >Reporter: Jiangjie Qin >Assignee: Jiangjie Qin > Fix For: 0.9.0.1 > > > Currently KafkaProducer will invoke callback.onComplete() if it receives an > ApiException during send(). This breaks the guarantee that callback will be > invoked in order. It seems ApiException in send() only comes from metadata > refresh. If so, we can probably simply throw it instead of invoking > callback(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
