[jira] [Updated] (KAFKA-1774) REPL and Shell Client for Admin Message RQ/RP
[ https://issues.apache.org/jira/browse/KAFKA-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1774: - Description: We should have a REPL we can work in and execute the commands with the arguments. With this we can do: ./kafka.sh --shell kafkaattach cluster -b localhost:9092; kafkadescribe topic sampleTopicNameForExample; the command line version can work like it does now so folks don't have to re-write all of their tooling. kafka.sh --topics --everything the same like kafka-topics.sh is kafka.sh --reassign --everything the same like kafka-reassign-partitions.sh is was: We should have a REPL we can work in and execute the commands with the arguments. With this we can do: ./kafka.sh --shell kafkaattach cluster -b localhost:9092; kafkadescribe topic sampleTopicNameForExample; the command line version can work like it does now so folks don't have to re-write all of their tooling. kafka.sh --topics --everything the same like kafka-topics.sh is kafka.sh --reassign --everything the same like kafka-reassign-partitions.sh is I think this part should reside in the client folder and be written in Go or Python (or Both) REPL and Shell Client for Admin Message RQ/RP - Key: KAFKA-1774 URL: https://issues.apache.org/jira/browse/KAFKA-1774 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Fix For: 0.8.3 We should have a REPL we can work in and execute the commands with the arguments. With this we can do: ./kafka.sh --shell kafkaattach cluster -b localhost:9092; kafkadescribe topic sampleTopicNameForExample; the command line version can work like it does now so folks don't have to re-write all of their tooling. kafka.sh --topics --everything the same like kafka-topics.sh is kafka.sh --reassign --everything the same like kafka-reassign-partitions.sh is -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot
[
https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-1664:
Assignee: Ashish Kumar Singh (was: Gwen Shapira)
Kafka does not properly parse multiple ZK nodes with non-root chroot
Key: KAFKA-1664
URL: https://issues.apache.org/jira/browse/KAFKA-1664
Project: Kafka
Issue Type: Bug
Components: clients
Reporter: Ricky Saltzer
Assignee: Ashish Kumar Singh
Priority: Minor
Labels: newbie
When using a non-root ZK directory for Kafka, if you specify multiple ZK
servers, Kafka does not seem to properly parse the connection string.
*Error*
{code}
[root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper
baelish-001.edh.cloudera.com:2181/kafka,baelish-002.edh.cloudera.com:2181/kafka,baelish-003.edh.cloudera.com:2181/kafka
--topic test-topic
[2014-10-01 15:31:04,629] ERROR Error processing message, stopping consumer:
(kafka.consumer.ConsoleConsumer$)
java.lang.IllegalArgumentException: Path length must be 0
at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:48)
at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:35)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:766)
at org.I0Itec.zkclient.ZkConnection.create(ZkConnection.java:87)
at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:308)
at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:304)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
at org.I0Itec.zkclient.ZkClient.create(ZkClient.java:304)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:213)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at kafka.utils.ZkUtils$.createParentPath(ZkUtils.scala:245)
at kafka.utils.ZkUtils$.createEphemeralPath(ZkUtils.scala:256)
at
kafka.utils.ZkUtils$.createEphemeralPathExpectConflict(ZkUtils.scala:268)
at
kafka.utils.ZkUtils$.createEphemeralPathExpectConflictHandleZKBug(ZkUtils.scala:306)
at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$registerConsumerInZK(ZookeeperConsumerConnector.scala:226)
at
kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.init(ZookeeperConsumerConnector.scala:755)
at
kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:196)
at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
{code}
*Working*
{code}
[root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper
baelish-001.edh.cloudera.com:2181/kafka --topic test-topic
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot
[
https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213899#comment-14213899
]
Gwen Shapira commented on KAFKA-1664:
-
Here you go, [~singhashish]. Have fun.
Kafka does not properly parse multiple ZK nodes with non-root chroot
Key: KAFKA-1664
URL: https://issues.apache.org/jira/browse/KAFKA-1664
Project: Kafka
Issue Type: Bug
Components: clients
Reporter: Ricky Saltzer
Assignee: Ashish Kumar Singh
Priority: Minor
Labels: newbie
When using a non-root ZK directory for Kafka, if you specify multiple ZK
servers, Kafka does not seem to properly parse the connection string.
*Error*
{code}
[root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper
baelish-001.edh.cloudera.com:2181/kafka,baelish-002.edh.cloudera.com:2181/kafka,baelish-003.edh.cloudera.com:2181/kafka
--topic test-topic
[2014-10-01 15:31:04,629] ERROR Error processing message, stopping consumer:
(kafka.consumer.ConsoleConsumer$)
java.lang.IllegalArgumentException: Path length must be 0
at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:48)
at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:35)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:766)
at org.I0Itec.zkclient.ZkConnection.create(ZkConnection.java:87)
at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:308)
at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:304)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
at org.I0Itec.zkclient.ZkClient.create(ZkClient.java:304)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:213)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at kafka.utils.ZkUtils$.createParentPath(ZkUtils.scala:245)
at kafka.utils.ZkUtils$.createEphemeralPath(ZkUtils.scala:256)
at
kafka.utils.ZkUtils$.createEphemeralPathExpectConflict(ZkUtils.scala:268)
at
kafka.utils.ZkUtils$.createEphemeralPathExpectConflictHandleZKBug(ZkUtils.scala:306)
at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$registerConsumerInZK(ZookeeperConsumerConnector.scala:226)
at
kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.init(ZookeeperConsumerConnector.scala:755)
at
kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:196)
at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
{code}
*Working*
{code}
[root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper
baelish-001.edh.cloudera.com:2181/kafka --topic test-topic
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1305) Controller can hang on controlled shutdown with auto leader balance enabled
[ https://issues.apache.org/jira/browse/KAFKA-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213960#comment-14213960 ] Jason Rosenberg commented on KAFKA-1305: Is it safe to say then, if we are not yet on 0.8.2 (e.g. still on 0.8.1.1), we should not enable automatic preferred leader election? Controller can hang on controlled shutdown with auto leader balance enabled --- Key: KAFKA-1305 URL: https://issues.apache.org/jira/browse/KAFKA-1305 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Sriharsha Chintalapani Priority: Blocker Fix For: 0.8.2, 0.9.0 Attachments: KAFKA-1305.patch, KAFKA-1305.patch, KAFKA-1305_2014-10-13_07:30:45.patch This is relatively easy to reproduce especially when doing a rolling bounce. What happened here is as follows: 1. The previous controller was bounced and broker 265 became the new controller. 2. I went on to do a controlled shutdown of broker 265 (the new controller). 3. In the mean time the automatically scheduled preferred replica leader election process started doing its thing and starts sending LeaderAndIsrRequests/UpdateMetadataRequests to itself (and other brokers). (t@113 below). 4. While that's happening, the controlled shutdown process on 265 succeeds and proceeds to deregister itself from ZooKeeper and shuts down the socket server. 5. (ReplicaStateMachine actually removes deregistered brokers from the controller channel manager's list of brokers to send requests to. However, that removal cannot take place (t@18 below) because preferred replica leader election task owns the controller lock.) 6. So the request thread to broker 265 gets into infinite retries. 7. The entire broker shutdown process is blocked on controller shutdown for the same reason (it needs to acquire the controller lock). Relevant portions from the thread-dump: Controller-265-to-broker-265-send-thread - Thread t@113 java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at kafka.controller.RequestSendThread$$anonfun$liftedTree1$1$1.apply$mcV$sp(ControllerChannelManager.scala:143) at kafka.utils.Utils$.swallow(Utils.scala:167) at kafka.utils.Logging$class.swallowWarn(Logging.scala:92) at kafka.utils.Utils$.swallowWarn(Utils.scala:46) at kafka.utils.Logging$class.swallow(Logging.scala:94) at kafka.utils.Utils$.swallow(Utils.scala:46) at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:143) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131) - locked java.lang.Object@6dbf14a7 at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) Locked ownable synchronizers: - None ... Thread-4 - Thread t@17 java.lang.Thread.State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@4836840 owned by: kafka-scheduler-0 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at kafka.utils.Utils$.inLock(Utils.scala:536) at kafka.controller.KafkaController.shutdown(KafkaController.scala:642) at kafka.server.KafkaServer$$anonfun$shutdown$9.apply$mcV$sp(KafkaServer.scala:242) at kafka.utils.Utils$.swallow(Utils.scala:167) at kafka.utils.Logging$class.swallowWarn(Logging.scala:92) at kafka.utils.Utils$.swallowWarn(Utils.scala:46) at kafka.utils.Logging$class.swallow(Logging.scala:94) at kafka.utils.Utils$.swallow(Utils.scala:46) at kafka.server.KafkaServer.shutdown(KafkaServer.scala:242) at kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:46) at kafka.Kafka$$anon$1.run(Kafka.scala:42) ... kafka-scheduler-0 - Thread t@117 java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1dc407fc at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at
Re: Review Request 27634: Patch for KAFKA-1667
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27634/ --- (Updated Nov. 16, 2014, 5:31 p.m.) Review request for kafka. Bugs: KAFKA-1667 https://issues.apache.org/jira/browse/KAFKA-1667 Repository: kafka Description (updated) --- KAFKA-1667 Fixed bugs in LogConfig. Added test and documentation KAFKA-1667 Updated tests to reflect new boolean property parsing logic KAFKA-1667 renamed methods to match naming convention KAFKA-1667 Added unit test to cover invalid configuration case KAFKA-1667 Strict UncleanLeaderElection property parsing KAFKA-1667 minor non-functional re-factoring Diffs (updated) - clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java c4cea2cc072f4db4ce014b63d226431d3766bef1 core/src/main/scala/kafka/admin/TopicCommand.scala 0b2735e7fc42ef9894bef1997b1f06a8ebee5439 core/src/main/scala/kafka/log/LogConfig.scala e48922a97727dd0b98f3ae630ebb0af3bef2373d core/src/main/scala/kafka/utils/Utils.scala 23aefb4715b177feae1d2f83e8b910653ea10c5f core/src/test/scala/kafka/log/LogConfigTest.scala PRE-CREATION core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala f44568cb25edf25db857415119018fd4c9922f61 Diff: https://reviews.apache.org/r/27634/diff/ Testing --- Thanks, Dmytro Kostiuchenko
[jira] [Updated] (KAFKA-1667) topic-level configuration not validated
[
https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmytro Kostiuchenko updated KAFKA-1667:
---
Attachment: KAFKA-1667_2014-11-16_18:31:34.patch
topic-level configuration not validated
Key: KAFKA-1667
URL: https://issues.apache.org/jira/browse/KAFKA-1667
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Ryan Berdeen
Labels: newbie
Attachments: KAFKA-1667_2014-11-05_19:43:53.patch,
KAFKA-1667_2014-11-06_17:10:14.patch, KAFKA-1667_2014-11-07_14:28:14.patch,
KAFKA-1667_2014-11-12_12:49:11.patch, KAFKA-1667_2014-11-16_18:31:34.patch
I was able to set the configuration for a topic to these invalid values:
{code}
Topic:topic-config-test PartitionCount:1ReplicationFactor:2
Configs:min.cleanable.dirty.ratio=-30.2,segment.bytes=-1,retention.ms=-12,cleanup.policy=lol
{code}
It seems that the values are saved as long as they are the correct type, but
are not validated like the corresponding broker-level properties.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1667) topic-level configuration not validated
[
https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213977#comment-14213977
]
Dmytro Kostiuchenko commented on KAFKA-1667:
Updated reviewboard https://reviews.apache.org/r/27634/diff/
against branch origin/trunk
topic-level configuration not validated
Key: KAFKA-1667
URL: https://issues.apache.org/jira/browse/KAFKA-1667
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Ryan Berdeen
Labels: newbie
Attachments: KAFKA-1667_2014-11-05_19:43:53.patch,
KAFKA-1667_2014-11-06_17:10:14.patch, KAFKA-1667_2014-11-07_14:28:14.patch,
KAFKA-1667_2014-11-12_12:49:11.patch, KAFKA-1667_2014-11-16_18:31:34.patch
I was able to set the configuration for a topic to these invalid values:
{code}
Topic:topic-config-test PartitionCount:1ReplicationFactor:2
Configs:min.cleanable.dirty.ratio=-30.2,segment.bytes=-1,retention.ms=-12,cleanup.policy=lol
{code}
It seems that the values are saved as long as they are the correct type, but
are not validated like the corresponding broker-level properties.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 27634: Patch for KAFKA-1667
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27634/ --- (Updated Nov. 16, 2014, 5:33 p.m.) Review request for kafka. Bugs: KAFKA-1667 https://issues.apache.org/jira/browse/KAFKA-1667 Repository: kafka Description --- KAFKA-1667 Fixed bugs in LogConfig. Added test and documentation KAFKA-1667 Updated tests to reflect new boolean property parsing logic KAFKA-1667 renamed methods to match naming convention KAFKA-1667 Added unit test to cover invalid configuration case KAFKA-1667 Strict UncleanLeaderElection property parsing KAFKA-1667 minor non-functional re-factoring Diffs (updated) - clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java c4cea2cc072f4db4ce014b63d226431d3766bef1 core/src/main/scala/kafka/admin/TopicCommand.scala 0b2735e7fc42ef9894bef1997b1f06a8ebee5439 core/src/main/scala/kafka/log/LogConfig.scala e48922a97727dd0b98f3ae630ebb0af3bef2373d core/src/main/scala/kafka/utils/Utils.scala 23aefb4715b177feae1d2f83e8b910653ea10c5f core/src/test/scala/kafka/log/LogConfigTest.scala PRE-CREATION core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala f44568cb25edf25db857415119018fd4c9922f61 Diff: https://reviews.apache.org/r/27634/diff/ Testing --- Thanks, Dmytro Kostiuchenko
Re: Review Request 27634: Patch for KAFKA-1667
On Nov. 15, 2014, 1:48 a.m., Jun Rao wrote: core/src/main/scala/kafka/log/LogConfig.scala, line 46 https://reviews.apache.org/r/27634/diff/5/?file=759553#file759553line46 This is actually a hard limit. All docs are just copied from javadoc. I've changed nothing, and thus haven't checked docs correctness. Fixed On Nov. 15, 2014, 1:48 a.m., Jun Rao wrote: core/src/main/scala/kafka/log/LogConfig.scala, lines 62-63 https://reviews.apache.org/r/27634/diff/5/?file=759553#file759553line62 This actually can be controlled at the topic level. So, I just removed the note about validation purposes. Is it ok? On Nov. 15, 2014, 1:48 a.m., Jun Rao wrote: core/src/test/scala/kafka/log/LogConfigTest.scala, line 52 https://reviews.apache.org/r/27634/diff/5/?file=759555#file759555line52 Does return skip the rest of the config names? Yes, it is. But it is a lambda that is called for each element of the set, so test actually validates every property. Also test will fail if new property with non-trivial validation (other than positive number) is added but is not covered in test. On Nov. 15, 2014, 1:48 a.m., Jun Rao wrote: core/src/main/scala/kafka/utils/Utils.scala, line 525 https://reviews.apache.org/r/27634/diff/5/?file=759554#file759554line525 Could we rename this to sth like overridesAndDefaults()? IMO, whatever we choose, it won't be clear from the name what the method does. Unless method is placed in some dedicated class like PropertiesUtils. Some context is required to understand, but it is not so complicated to bother anyway. So, I would leave it as it is. Please confirm that renaming to overridesAndDefaults is necessary and I will do that. - Dmytro --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27634/#review61581 --- On Nov. 16, 2014, 5:33 p.m., Dmytro Kostiuchenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27634/ --- (Updated Nov. 16, 2014, 5:33 p.m.) Review request for kafka. Bugs: KAFKA-1667 https://issues.apache.org/jira/browse/KAFKA-1667 Repository: kafka Description --- KAFKA-1667 Fixed bugs in LogConfig. Added test and documentation KAFKA-1667 Updated tests to reflect new boolean property parsing logic KAFKA-1667 renamed methods to match naming convention KAFKA-1667 Added unit test to cover invalid configuration case KAFKA-1667 Strict UncleanLeaderElection property parsing KAFKA-1667 minor non-functional re-factoring Diffs - clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java c4cea2cc072f4db4ce014b63d226431d3766bef1 core/src/main/scala/kafka/admin/TopicCommand.scala 0b2735e7fc42ef9894bef1997b1f06a8ebee5439 core/src/main/scala/kafka/log/LogConfig.scala e48922a97727dd0b98f3ae630ebb0af3bef2373d core/src/main/scala/kafka/utils/Utils.scala 23aefb4715b177feae1d2f83e8b910653ea10c5f core/src/test/scala/kafka/log/LogConfigTest.scala PRE-CREATION core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala f44568cb25edf25db857415119018fd4c9922f61 Diff: https://reviews.apache.org/r/27634/diff/ Testing --- Thanks, Dmytro Kostiuchenko
[jira] [Updated] (KAFKA-1667) topic-level configuration not validated
[
https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmytro Kostiuchenko updated KAFKA-1667:
---
Attachment: KAFKA-1667_2014-11-16_18:33:10.patch
topic-level configuration not validated
Key: KAFKA-1667
URL: https://issues.apache.org/jira/browse/KAFKA-1667
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Ryan Berdeen
Labels: newbie
Attachments: KAFKA-1667_2014-11-05_19:43:53.patch,
KAFKA-1667_2014-11-06_17:10:14.patch, KAFKA-1667_2014-11-07_14:28:14.patch,
KAFKA-1667_2014-11-12_12:49:11.patch, KAFKA-1667_2014-11-16_18:31:34.patch,
KAFKA-1667_2014-11-16_18:33:10.patch
I was able to set the configuration for a topic to these invalid values:
{code}
Topic:topic-config-test PartitionCount:1ReplicationFactor:2
Configs:min.cleanable.dirty.ratio=-30.2,segment.bytes=-1,retention.ms=-12,cleanup.policy=lol
{code}
It seems that the values are saved as long as they are the correct type, but
are not validated like the corresponding broker-level properties.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1667) topic-level configuration not validated
[
https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213978#comment-14213978
]
Dmytro Kostiuchenko commented on KAFKA-1667:
Updated reviewboard https://reviews.apache.org/r/27634/diff/
against branch origin/trunk
topic-level configuration not validated
Key: KAFKA-1667
URL: https://issues.apache.org/jira/browse/KAFKA-1667
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Ryan Berdeen
Labels: newbie
Attachments: KAFKA-1667_2014-11-05_19:43:53.patch,
KAFKA-1667_2014-11-06_17:10:14.patch, KAFKA-1667_2014-11-07_14:28:14.patch,
KAFKA-1667_2014-11-12_12:49:11.patch, KAFKA-1667_2014-11-16_18:31:34.patch,
KAFKA-1667_2014-11-16_18:33:10.patch
I was able to set the configuration for a topic to these invalid values:
{code}
Topic:topic-config-test PartitionCount:1ReplicationFactor:2
Configs:min.cleanable.dirty.ratio=-30.2,segment.bytes=-1,retention.ms=-12,cleanup.policy=lol
{code}
It seems that the values are saved as long as they are the correct type, but
are not validated like the corresponding broker-level properties.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1305) Controller can hang on controlled shutdown with auto leader balance enabled
[ https://issues.apache.org/jira/browse/KAFKA-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213997#comment-14213997 ] Sriharsha Chintalapani commented on KAFKA-1305: --- [~jbrosenb...@gmail.com] The fix is in config. So if you are using 0.8.1.1 you can enable automatic preferred leader election but make sure you set controller.message.queue.size to Int.MaxValue by default this is set to 10 in 0.8.1.1. Controller can hang on controlled shutdown with auto leader balance enabled --- Key: KAFKA-1305 URL: https://issues.apache.org/jira/browse/KAFKA-1305 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Sriharsha Chintalapani Priority: Blocker Fix For: 0.8.2, 0.9.0 Attachments: KAFKA-1305.patch, KAFKA-1305.patch, KAFKA-1305_2014-10-13_07:30:45.patch This is relatively easy to reproduce especially when doing a rolling bounce. What happened here is as follows: 1. The previous controller was bounced and broker 265 became the new controller. 2. I went on to do a controlled shutdown of broker 265 (the new controller). 3. In the mean time the automatically scheduled preferred replica leader election process started doing its thing and starts sending LeaderAndIsrRequests/UpdateMetadataRequests to itself (and other brokers). (t@113 below). 4. While that's happening, the controlled shutdown process on 265 succeeds and proceeds to deregister itself from ZooKeeper and shuts down the socket server. 5. (ReplicaStateMachine actually removes deregistered brokers from the controller channel manager's list of brokers to send requests to. However, that removal cannot take place (t@18 below) because preferred replica leader election task owns the controller lock.) 6. So the request thread to broker 265 gets into infinite retries. 7. The entire broker shutdown process is blocked on controller shutdown for the same reason (it needs to acquire the controller lock). Relevant portions from the thread-dump: Controller-265-to-broker-265-send-thread - Thread t@113 java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at kafka.controller.RequestSendThread$$anonfun$liftedTree1$1$1.apply$mcV$sp(ControllerChannelManager.scala:143) at kafka.utils.Utils$.swallow(Utils.scala:167) at kafka.utils.Logging$class.swallowWarn(Logging.scala:92) at kafka.utils.Utils$.swallowWarn(Utils.scala:46) at kafka.utils.Logging$class.swallow(Logging.scala:94) at kafka.utils.Utils$.swallow(Utils.scala:46) at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:143) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131) - locked java.lang.Object@6dbf14a7 at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) Locked ownable synchronizers: - None ... Thread-4 - Thread t@17 java.lang.Thread.State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@4836840 owned by: kafka-scheduler-0 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at kafka.utils.Utils$.inLock(Utils.scala:536) at kafka.controller.KafkaController.shutdown(KafkaController.scala:642) at kafka.server.KafkaServer$$anonfun$shutdown$9.apply$mcV$sp(KafkaServer.scala:242) at kafka.utils.Utils$.swallow(Utils.scala:167) at kafka.utils.Logging$class.swallowWarn(Logging.scala:92) at kafka.utils.Utils$.swallowWarn(Utils.scala:46) at kafka.utils.Logging$class.swallow(Logging.scala:94) at kafka.utils.Utils$.swallow(Utils.scala:46) at kafka.server.KafkaServer.shutdown(KafkaServer.scala:242) at kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:46) at kafka.Kafka$$anon$1.run(Kafka.scala:42) ... kafka-scheduler-0 - Thread t@117 java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1dc407fc at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at
Review Request 28106: Patch for KAFKA-1664
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28106/ --- Review request for kafka. Bugs: KAFKA-1664 https://issues.apache.org/jira/browse/KAFKA-1664 Repository: kafka Description --- KAFKA-1664: Kafka does not properly parse multiple ZK nodes with non-root chroot Diffs - .reviewboardrc 5e8d67014c01db52ab332d3dfb090a70fbb716d2 core/src/main/scala/kafka/utils/ZkUtils.scala 56e3e88e0cc6d917b0ffd1254e173295c1c4aabd Diff: https://reviews.apache.org/r/28106/diff/ Testing --- Thanks, Ashish Singh
Re: Review Request 28106: Patch for KAFKA-1664
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28106/ --- (Updated Nov. 16, 2014, 7:41 p.m.) Review request for kafka. Bugs: KAFKA-1664 https://issues.apache.org/jira/browse/KAFKA-1664 Repository: kafka Description --- KAFKA-1664: Kafka does not properly parse multiple ZK nodes with non-root chroot Diffs (updated) - .reviewboardrc 5e8d67014c01db52ab332d3dfb090a70fbb716d2 core/src/main/scala/kafka/utils/ZkUtils.scala 56e3e88e0cc6d917b0ffd1254e173295c1c4aabd Diff: https://reviews.apache.org/r/28106/diff/ Testing --- Thanks, Ashish Singh
Review Request 28108: KAFKA-1664: Kafka does not properly parse multiple ZK nodes with non-root chroot
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28108/ --- Review request for kafka. Repository: kafka Description --- KAFKA-1664: Kafka does not properly parse multiple ZK nodes with non-root chroot Diffs - core/src/main/scala/kafka/utils/ZkUtils.scala 56e3e88e0cc6d917b0ffd1254e173295c1c4aabd Diff: https://reviews.apache.org/r/28108/diff/ Testing --- Tested with and without the fix. Thanks, Ashish Singh
Re: Review Request 28108: KAFKA-1664: Kafka does not properly parse multiple ZK nodes with non-root chroot
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28108/ --- (Updated Nov. 16, 2014, 7:49 p.m.) Review request for kafka. Repository: kafka Description --- KAFKA-1664: Kafka does not properly parse multiple ZK nodes with non-root chroot Diffs - core/src/main/scala/kafka/utils/ZkUtils.scala 56e3e88e0cc6d917b0ffd1254e173295c1c4aabd Diff: https://reviews.apache.org/r/28108/diff/ Testing --- Tested with and without the fix. Thanks, Ashish Singh
[jira] [Updated] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot
[
https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Kumar Singh updated KAFKA-1664:
--
Status: Patch Available (was: Open)
Kafka does not properly parse multiple ZK nodes with non-root chroot
Key: KAFKA-1664
URL: https://issues.apache.org/jira/browse/KAFKA-1664
Project: Kafka
Issue Type: Bug
Components: clients
Reporter: Ricky Saltzer
Assignee: Ashish Kumar Singh
Priority: Minor
Labels: newbie
Attachments: KAFKA-1664.patch
When using a non-root ZK directory for Kafka, if you specify multiple ZK
servers, Kafka does not seem to properly parse the connection string.
*Error*
{code}
[root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper
baelish-001.edh.cloudera.com:2181/kafka,baelish-002.edh.cloudera.com:2181/kafka,baelish-003.edh.cloudera.com:2181/kafka
--topic test-topic
[2014-10-01 15:31:04,629] ERROR Error processing message, stopping consumer:
(kafka.consumer.ConsoleConsumer$)
java.lang.IllegalArgumentException: Path length must be 0
at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:48)
at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:35)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:766)
at org.I0Itec.zkclient.ZkConnection.create(ZkConnection.java:87)
at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:308)
at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:304)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
at org.I0Itec.zkclient.ZkClient.create(ZkClient.java:304)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:213)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at kafka.utils.ZkUtils$.createParentPath(ZkUtils.scala:245)
at kafka.utils.ZkUtils$.createEphemeralPath(ZkUtils.scala:256)
at
kafka.utils.ZkUtils$.createEphemeralPathExpectConflict(ZkUtils.scala:268)
at
kafka.utils.ZkUtils$.createEphemeralPathExpectConflictHandleZKBug(ZkUtils.scala:306)
at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$registerConsumerInZK(ZookeeperConsumerConnector.scala:226)
at
kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.init(ZookeeperConsumerConnector.scala:755)
at
kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:196)
at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
{code}
*Working*
{code}
[root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper
baelish-001.edh.cloudera.com:2181/kafka --topic test-topic
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot
[
https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Kumar Singh updated KAFKA-1664:
--
Attachment: KAFKA-1664.patch
RB: https://reviews.apache.org/r/28108/
Kafka does not properly parse multiple ZK nodes with non-root chroot
Key: KAFKA-1664
URL: https://issues.apache.org/jira/browse/KAFKA-1664
Project: Kafka
Issue Type: Bug
Components: clients
Reporter: Ricky Saltzer
Assignee: Ashish Kumar Singh
Priority: Minor
Labels: newbie
Attachments: KAFKA-1664.patch
When using a non-root ZK directory for Kafka, if you specify multiple ZK
servers, Kafka does not seem to properly parse the connection string.
*Error*
{code}
[root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper
baelish-001.edh.cloudera.com:2181/kafka,baelish-002.edh.cloudera.com:2181/kafka,baelish-003.edh.cloudera.com:2181/kafka
--topic test-topic
[2014-10-01 15:31:04,629] ERROR Error processing message, stopping consumer:
(kafka.consumer.ConsoleConsumer$)
java.lang.IllegalArgumentException: Path length must be 0
at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:48)
at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:35)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:766)
at org.I0Itec.zkclient.ZkConnection.create(ZkConnection.java:87)
at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:308)
at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:304)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
at org.I0Itec.zkclient.ZkClient.create(ZkClient.java:304)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:213)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
at kafka.utils.ZkUtils$.createParentPath(ZkUtils.scala:245)
at kafka.utils.ZkUtils$.createEphemeralPath(ZkUtils.scala:256)
at
kafka.utils.ZkUtils$.createEphemeralPathExpectConflict(ZkUtils.scala:268)
at
kafka.utils.ZkUtils$.createEphemeralPathExpectConflictHandleZKBug(ZkUtils.scala:306)
at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$registerConsumerInZK(ZookeeperConsumerConnector.scala:226)
at
kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.init(ZookeeperConsumerConnector.scala:755)
at
kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:196)
at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
{code}
*Working*
{code}
[root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper
baelish-001.edh.cloudera.com:2181/kafka --topic test-topic
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
I would like to subscribe
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214296#comment-14214296 ] Manikumar Reddy commented on KAFKA-1173: +1 for this patch. It is really useful for creating development test beds. Using Vagrant to get up and running with Apache Kafka - Key: KAFKA-1173 URL: https://issues.apache.org/jira/browse/KAFKA-1173 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Assignee: Ewen Cheslack-Postava Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch Vagrant has been getting a lot of pickup in the tech communities. I have found it very useful for development and testing and working with a few clients now using it to help virtualize their environments in repeatable ways. Using Vagrant to get up and running. For 0.8.0 I have a patch on github https://github.com/stealthly/kafka 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) 2) Install Virtual Box [https://www.virtualbox.org/](https://www.virtualbox.org/) In the main kafka folder 1) ./sbt update 2) ./sbt package 3) ./sbt assembly-package-dependency 4) vagrant up once this is done * Zookeeper will be running 192.168.50.5 * Broker 1 on 192.168.50.10 * Broker 2 on 192.168.50.20 * Broker 3 on 192.168.50.30 When you are all up and running you will be back at a command brompt. If you want you can login to the machines using vagrant shh machineName but you don't need to. You can access the brokers and zookeeper by their IP e.g. bin/kafka-console-producer.sh --broker-list 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
