[
https://issues.apache.org/jira/browse/KAFKA-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153343#comment-14153343
]
Sriharsha Chintalapani commented on KAFKA-1558:
-----------------------------------------------
[~nehanarkhede] [~junrao] This seems to be related to the earlier issue I
reported
https://issues.apache.org/jira/browse/KAFKA-1558?focusedCommentId=14142342&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14142342
possible issue is related to old controller doesn't shutdown properly when a
new controller is elected.
I ran a simple test with 1000 topics in 5 broker cluster . This is without
consumers or producers running and no delete topic command issued.
After a soft failure old controller log shows
[2014-09-30 15:59:53,398] INFO [SessionExpirationListener on 1], ZK expired;
shut down all controller components and try to re-elect
(kafka.controller.KafkaController$SessionExpirationListener)
[2014-09-30 15:59:53,400] INFO [delete-topics-thread-1], Shutting down
(kafka.controller.TopicDeletionManager$DeleteTopicsThread)
It stops there and the server.log goes on with
[2014-09-30 16:17:36,649] INFO Partition [my-topic-634,0] on broker 1:
Shrinking ISR for partition [my-topic-634,0] from 1,3,4 to 1
(kafka.cluster.Partition)
[2014-09-30 16:17:36,653] INFO Partition [my-topic-634,0] on broker 1: Cached
zkVersion [0] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)
[2014-09-30 16:17:36,653] INFO Partition [my-topic-374,1] on broker 1:
Shrinking ISR for partition [my-topic-374,1] from 1,2,3 to 1
(kafka.cluster.Partition)
[2014-09-30 16:17:36,656] INFO Partition [my-topic-374,1] on broker 1: Cached
zkVersion [0] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)
[2014-09-30 16:17:36,657] INFO Partition [my-topic-549,2] on broker 1:
Shrinking ISR for partition [my-topic-549,2] from 1,2,3 to 1
(kafka.cluster.Partition)
I tried reproduce this in a 3 node cluster in vms with 200 topics and with or
without producers & consumers running.
But here old controller shutdown goes through fine.
[2014-09-30 14:50:55,193] INFO [SessionExpirationListener on 3], ZK expired;
shut down all controller components and try to re-elect
(kafka.controller.KafkaController$SessionExpirationListener)
[2014-09-30 14:50:55,196] INFO [delete-topics-thread-3], Shutting down
(kafka.controller.TopicDeletionManager$DeleteTopicsThread)
[2014-09-30 14:50:55,200] INFO [delete-topics-thread-3], Stopped
(kafka.controller.TopicDeletionManager$DeleteTopicsThread)
[2014-09-30 14:50:55,200] INFO [delete-topics-thread-3], Shutdown completed
(kafka.controller.TopicDeletionManager$DeleteTopicsThread)
[2014-09-30 14:50:55,202] INFO [Partition state machine on Controller 3]:
Stopped partition state machine (kafka.controller.PartitionStateMachine)
[2014-09-30 14:50:55,202] INFO [Replica state machine on controller 3]: Stopped
replica state machine (kafka.controller.ReplicaStateMachine)
[2014-09-30 14:50:55,202] INFO [Controller-3-to-broker-2-send-thread], Shutting
down (kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,202] INFO [Controller-3-to-broker-2-send-thread], Stopped
(kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,202] INFO [Controller-3-to-broker-2-send-thread], Shutdown
completed (kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,202] INFO [Controller-3-to-broker-1-send-thread], Shutting
down (kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,202] INFO [Controller-3-to-broker-1-send-thread], Stopped
(kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,203] INFO [Controller-3-to-broker-1-send-thread], Shutdown
completed (kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,203] INFO [Controller-3-to-broker-3-send-thread], Shutting
down (kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,203] INFO [Controller-3-to-broker-3-send-thread], Stopped
(kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,203] INFO [Controller-3-to-broker-3-send-thread], Shutdown
completed (kafka.controller.RequestSendThread)
Regarding TopicDeletionManager shouldn't we stop if one of the replicas are
offline or atleast have configurable number of retries for topic deletion?
> AdminUtils.deleteTopic does not work
> ------------------------------------
>
> Key: KAFKA-1558
> URL: https://issues.apache.org/jira/browse/KAFKA-1558
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8.1.1
> Reporter: Henning Schmiedehausen
> Assignee: Sriharsha Chintalapani
> Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: kafka-thread-dump.log
>
>
> the AdminUtils:.deleteTopic method is implemented as
> {code}
> def deleteTopic(zkClient: ZkClient, topic: String) {
> ZkUtils.createPersistentPath(zkClient,
> ZkUtils.getDeleteTopicPath(topic))
> }
> {code}
> but the DeleteTopicCommand actually does
> {code}
> zkClient = new ZkClient(zkConnect, 30000, 30000, ZKStringSerializer)
> zkClient.deleteRecursive(ZkUtils.getTopicPath(topic))
> {code}
> so I guess, that the 'createPersistentPath' above should actually be
> {code}
> def deleteTopic(zkClient: ZkClient, topic: String) {
> ZkUtils.deletePathRecursive(zkClient, ZkUtils.getTopicPath(topic))
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
