One other clue, is that although those topics still show up under /brokers/topics/<topicname>/ they contain empty sub-nodes.....So zk knows they don't exist on any broker. So maybe that's the issue. Brokers are successfully removing themselves as serving a topic, but the topic itself remains. The consumers then just see the topics (even with no brokers listed) and continue to watch the topic.
Jason On Mon, Mar 18, 2013 at 1:37 PM, Jason Rosenberg <j...@squareup.com> wrote: > Thanks Jun, > > I have nothing listed under /brokers/topics/deletedtopic, etc....so that > doesn't appear to be the issue. > > I will try removing now the unwanted topics under /brokers/topics/. > > Jason > > > On Mon, Mar 18, 2013 at 9:31 AM, Jun Rao <jun...@gmail.com> wrote: > >> Jason, >> >> This is mainly a problem that we don't have a formal way of deleting a >> topic in 0.7, which we are trying to fix in 0.8. >> >> The extra watchers on those deleted topics are likely registered by the >> producers. They should be gone once /brokers/topics/deletedtopic are >> removed from ZK. >> >> You probably don't need to clean /consumers/<appname>/owners. >> >> Thanks, >> >> Jun >> >> >> On Sun, Mar 17, 2013 at 11:11 PM, Jason Rosenberg <j...@squareup.com> >> wrote: >> >> > Jun, >> > >> > There are indeed no nodes under /brokers/topics/deletedtopic >> > >> > Also, do I need to remove the deleted apps from the >> > /consumers/<appname>/owners path? >> > >> > So, should this be considered a bug, or not? Essentially, each broker >> > knows what topics it has, but the info seems to have not propagated to >> zk >> > at all? Is that right? >> > >> > Although from the telnet interface, the removed topics don't show up >> under >> > a 'dump' command, but they do under wchc, etc. What's that about? >> > >> > Should the original procedure have been to: (1) stop all consumers, then >> > (2) remove the topic files on the brokers, then (3) restart the brokers, >> > then (4) restart the consumers? >> > >> > The consumers have been up and down since the topic files were removed, >> but >> > just not ever at the same time the brokers were down.... >> > >> > Jason >> > >> > >> > On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <jun...@gmail.com> wrote: >> > >> > > Jason, >> > > >> > > Use zkCli.sh to see if there is any child node in ZK under >> > > /brokers/topics/deletedtopic. You should see none since you already >> > removed >> > > those topics from brokers' log. If so, you can manually remove those >> > paths >> > > in ZK using zkCli.sh (stop the consumers first). After the consumers >> > > restart, you should not see the the deleted topics in the log of the >> > > consumers any more. >> > > >> > > consumer.ZookeeperConsumerConnector - >> > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb >> Topics to >> > > consume = List(<a bazillion topics, most of which have been deleted >> > listed >> > > here>) >> > > >> > > Thanks, >> > > >> > > Jun >> > > >> > > On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <j...@squareup.com> >> > > wrote: >> > > >> > > > Hi Jun, yes, we're using zk-based producers. Producers have all >> been >> > > > restarted (they get cycled many times a day as well). >> > > > >> > > > The specific producers that generated all those bogus topics have >> > > > definitely been restarted, etc. >> > > > >> > > > Using zkCli.sh, I do see all the deleted topics, e.g.: >> > > > >> > > > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics >> > > > [<1000's of deleted topics>,<also all the remaining valid topics>] >> > > > >> > > > I see also all the deleted topics under /consumers/<appname>/owners/ >> > > > >> > > > Jason >> > > > >> > > > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <jun...@gmail.com> wrote: >> > > > >> > > > > Are you using ZK-based producer? If so, those watches could be >> left >> > by >> > > > the >> > > > > producers if they haven't been restarted. Could you also use >> zkCli.sh >> > > to >> > > > > see if deleted topics are there in ZK? >> > > > > >> > > > > Thanks, >> > > > > >> > > > > Jun >> > > > > >> > > > > On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg < >> j...@squareup.com> >> > > > wrote: >> > > > > >> > > > > > Jun, >> > > > > > >> > > > > > So, I connected to zookeeper just using telnet, and using the 4 >> > > letter >> > > > > > commands. >> > > > > > >> > > > > > If I do a dump: >> > > > > > >> > > > > > I do not see anything but valid topics, and valid >> consumer/owners >> > > > > mappings. >> > > > > > >> > > > > > If I check watches, I see all the 1000's of bogus topics, e.c.: >> > > > > > >> > > > > > wchc: >> > > > > > .... >> > > > > > /brokers/topics/<obsolete-topic> >> > > > > > .... >> > > > > > >> > > > > > or >> > > > > > >> > > > > > wchp >> > > > > > .... >> > > > > > /brokers/topics/<obsolete-topic>. >> > > > > > 0x383d6fe1b83f0008 >> > > > > > 0x383d6fe1b83f0009 >> > > > > > 0x7c3d6f9020d00041 >> > > > > > 0x833d6fe2826e0000 >> > > > > > 0x383d6fe1b83f000a >> > > > > > .... >> > > > > > >> > > > > > The kafka and zookeeper servers, as well as the consumer apps, >> are >> > > > rolled >> > > > > > several times a day, automatically. >> > > > > > >> > > > > > I'm not sure how to remove the watches, is that what I need to >> do? >> > > > > > >> > > > > > Since this is a highly available system, can I bring down each >> zk >> > > > server >> > > > > 1 >> > > > > > at a time, and clear the bogus topics (or watches)? How do I do >> > > this? >> > > > > > >> > > > > > Jason >> > > > > > >> > > > > > >> > > > > > >> > > > > > On Fri, Mar 15, 2013 at 8:17 AM, Jun Rao <jun...@gmail.com> >> wrote: >> > > > > > >> > > > > > > Could you check if the following path for a deleted topic >> exists >> > in >> > > > ZK? >> > > > > > It >> > > > > > > should have no children. >> > > > > > > >> > > > > > > /brokers/topics/[topic] >> > > > > > > >> > > > > > > If this is the case, try manually removing those paths from ZK >> > > (when >> > > > > the >> > > > > > > brokers and the consumers are down). >> > > > > > > >> > > > > > > Thanks, >> > > > > > > >> > > > > > > Jun >> > > > > > > >> > > > > > > On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg < >> > j...@squareup.com >> > > > >> > > > > > wrote: >> > > > > > > >> > > > > > > > Hi Neha, >> > > > > > > > >> > > > > > > > So I did this, but I still see the full list of topics >> (most of >> > > > which >> > > > > > > have >> > > > > > > > been deleted), in the consumer logs, e.g.: >> > > > > > > > >> > > > > > > > consumer.ZookeeperConsumerConnector - >> > > > > > > > >> samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb >> > > > > Topics >> > > > > > to >> > > > > > > > consume = List(<a bazillion topics, most of which have been >> > > deleted >> > > > > > > listed >> > > > > > > > here>) >> > > > > > > > >> > > > > > > > I select topics using the white list topic selector (using a >> > > > > wild-card >> > > > > > > > pattern, etc.). >> > > > > > > > >> > > > > > > > Thoughts? >> > > > > > > > >> > > > > > > > Does zookeeper keep separate info for the consumer, that >> needs >> > to >> > > > be >> > > > > > > > removed? >> > > > > > > > >> > > > > > > > Jason >> > > > > > > > >> > > > > > > > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede < >> > > > > > neha.narkh...@gmail.com >> > > > > > > > >wrote: >> > > > > > > > >> > > > > > > > > In Kafka 0.7.2, data log directory delete and then rolling >> > > > restart >> > > > > > will >> > > > > > > > > work to get rid of the topic completely from zookeeper as >> > well >> > > as >> > > > > the >> > > > > > > > > broker. >> > > > > > > > > This is because the broker registers the topics and >> > partitions >> > > > from >> > > > > > its >> > > > > > > > > data directory into zookeeper on startup. >> > > > > > > > > >> > > > > > > > > Thanks, >> > > > > > > > > Neha >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg < >> > > > j...@squareup.com> >> > > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Thanks Neha, >> > > > > > > > > > >> > > > > > > > > > So are you saying that on 0.7.2, to delete a topic I >> need >> > > only >> > > > > > remove >> > > > > > > > > it's >> > > > > > > > > > data log directory from each broker, and the restart the >> > > > brokers? >> > > > > > Is >> > > > > > > > it >> > > > > > > > > ok >> > > > > > > > > > if it's a rolling restart? >> > > > > > > > > > >> > > > > > > > > > For some reason I thought I also had to do something on >> > > > zookeeper >> > > > > > > > servers >> > > > > > > > > > too, to remove the topics. Is that not true? >> > > > > > > > > > >> > > > > > > > > > Jason >> > > > > > > > > > >> > > > > > > > > > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede < >> > > > > > > neha.narkh...@gmail.com >> > > > > > > > > > >wrote: >> > > > > > > > > > >> > > > > > > > > > > >> The topics get automagically created if they don't >> > > exist. >> > > > > > Will >> > > > > > > > > > > that still exist in 0.8? >> > > > > > > > > > > >> > > > > > > > > > > In 0.8, this is controlled by the >> > auto.create.topics.enable >> > > > > > config >> > > > > > > on >> > > > > > > > > the >> > > > > > > > > > > brokers. If this is set to true, topics will be >> created >> > > when >> > > > a >> > > > > > > topic >> > > > > > > > > > > metadata request is sent for a new topic. This >> feature is >> > > > > > provided >> > > > > > > to >> > > > > > > > > aid >> > > > > > > > > > > with the 0.7->0.8 migration tool and the mirror maker >> > > tools. >> > > > > > These >> > > > > > > > > tools >> > > > > > > > > > > copy data for almost all topics from one cluster to >> the >> > > > other, >> > > > > so >> > > > > > > > auto >> > > > > > > > > > > creating topics makes it much easier to migrate data >> > > between >> > > > > > > > clusters. >> > > > > > > > > > > >> > > > > > > > > > > >> What's the procedure for deleting topics in kafka >> > 0.7.2? >> > > > > > > > > > > >> > > > > > > > > > > There is no official way to delete a topic. You have >> to >> > > > delete >> > > > > > the >> > > > > > > > data >> > > > > > > > > > > directory and bounce the brokers to take note of that. >> > > > > > > > > > > >> > > > > > > > > > > Thanks, >> > > > > > > > > > > Neha >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg < >> > > > > > j...@squareup.com> >> > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Thanks Neha, >> > > > > > > > > > > > >> > > > > > > > > > > > I added a comment to that ticket. >> > > > > > > > > > > > >> > > > > > > > > > > > Another question, I'm not sure if the behavior is >> > > different >> > > > > in >> > > > > > > 0.8 >> > > > > > > > > from >> > > > > > > > > > > > 0.7. But currently, I'm not predeclaring topics, >> > before >> > > > > > sending >> > > > > > > > > > messages >> > > > > > > > > > > > to topics. The topics get automagically created if >> > they >> > > > > don't >> > > > > > > > exist. >> > > > > > > > > > > Will >> > > > > > > > > > > > that still exist in 0.8? If so, it should also be >> > > > automatic >> > > > > > > > (using a >> > > > > > > > > > > > default expiration for all such topics), that they >> > > > > > automagically >> > > > > > > go >> > > > > > > > > > away >> > > > > > > > > > > > after no longer being in use. >> > > > > > > > > > > > >> > > > > > > > > > > > What's the procedure for deleting topics in kafka >> > 0.7.2? >> > > > > > > > > > > > >> > > > > > > > > > > > Jason >> > > > > > > > > > > > >> > > > > > > > > > > > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede < >> > > > > > > > > neha.narkh...@gmail.com >> > > > > > > > > > > > >wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > We plan to include support for deleting topics >> before >> > > the >> > > > > > > > official >> > > > > > > > > > > > release. >> > > > > > > > > > > > > The JIRA tracking this feature is >> > > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-330 >> > > > > > > > > > > > > >> > > > > > > > > > > > > This is mainly an admin tool and is proposed to be >> > run >> > > > > > > manually. >> > > > > > > > I >> > > > > > > > > > > think >> > > > > > > > > > > > > your suggestion is to set some sort of expiration >> > time >> > > on >> > > > > > > topics >> > > > > > > > so >> > > > > > > > > > > when >> > > > > > > > > > > > > all data gets garbage collected and the expiration >> > time >> > > > > > lapses, >> > > > > > > > the >> > > > > > > > > > > topic >> > > > > > > > > > > > > is automatically deleted. >> > > > > > > > > > > > > >> > > > > > > > > > > > > For this expiration to work, we need some >> mechanism >> > of >> > > > > > knowing >> > > > > > > > the >> > > > > > > > > > size >> > > > > > > > > > > > of >> > > > > > > > > > > > > the data logs for that topic on all brokers. And >> only >> > > > when >> > > > > > its >> > > > > > > > > empty >> > > > > > > > > > > for >> > > > > > > > > > > > > all partitions of that topic, we delete it. This >> > > requires >> > > > > > some >> > > > > > > > API >> > > > > > > > > > that >> > > > > > > > > > > > > will expose the data size of a topic on a broker, >> > which >> > > > we >> > > > > > > don't >> > > > > > > > > have >> > > > > > > > > > > > right >> > > > > > > > > > > > > now. It might be worth thinking about it. Do you >> mind >> > > > > posting >> > > > > > > > your >> > > > > > > > > > > > > suggestions for this tool on that JIRA ? >> > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks, >> > > > > > > > > > > > > Neha >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg < >> > > > > > > > j...@squareup.com> >> > > > > > > > > > > > wrote: >> > > > > > > > > > > > > >> > > > > > > > > > > > > > I suspect this is not currently supported, but >> it >> > > seems >> > > > > to >> > > > > > be >> > > > > > > > for >> > > > > > > > > > us >> > > > > > > > > > > a >> > > > > > > > > > > > > real >> > > > > > > > > > > > > > use case. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > If we have a topic that is no longer receiving >> > > > messages, >> > > > > > and >> > > > > > > > all >> > > > > > > > > > > > messages >> > > > > > > > > > > > > > have been removed from the brokers, after the >> > > > > > > > log_retention_hours >> > > > > > > > > > has >> > > > > > > > > > > > > > expired, I'd love to have it then automatically >> > > remove >> > > > > the >> > > > > > > > topic >> > > > > > > > > > from >> > > > > > > > > > > > > > kafka. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > If subsequently a new message arrives for that >> > topic, >> > > > it >> > > > > > > could >> > > > > > > > > just >> > > > > > > > > > > > > > recreate the topic as originally, and continue >> on. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Is this easy to implement? >> > > > > > > > > > > > > > Has it been requested? >> > > > > > > > > > > > > > Does it make sense? >> > > > > > > > > > > > > > Should I file a jira? >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Currently, we had a test client that >> inadvertently >> > > > > created >> > > > > > > > 1000's >> > > > > > > > > > of >> > > > > > > > > > > > > > topics. The job of deleting all these appears >> to >> > not >> > > > be >> > > > > a >> > > > > > > > > > cumbersome >> > > > > > > > > > > > > > manual process (are there any quick links to the >> > > > > > recommended >> > > > > > > > way >> > > > > > > > > > for >> > > > > > > > > > > > > doing >> > > > > > > > > > > > > > this?). >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > I am using 0.7.2, but will eventually upgrade to >> > 0.8, >> > > > > once >> > > > > > > it's >> > > > > > > > > > > > released >> > > > > > > > > > > > > to >> > > > > > > > > > > > > > beta.... >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Thanks, >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Jason >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >
