FWIW, Here’s the output of kafka-mesos.sh scheduler: — [root@zk01 kafka-mesos]# ./kafka-mesos.sh scheduler Loading config defaults from kafka-mesos.properties 2016-04-13 15:34:29,168 [main] INFO ly.stealth.mesos.kafka.Scheduler$ - Starting Scheduler$: debug: true, storage: zk:/mesos-kafka mesos: master=zk01.something.com:5050,zk02.something.com:5050,zk03.something.com:5050, user=marathon, principal=<none>, secret=<none> framework: name=kafka, role=*, timeout=30d api: http://zk01.something.com:7000, bind-address: <all>, zk: zk01.something.com:2181,zk02.something.com:2181,zk03.something.com:2181/kafka, jre: 2016-04-13 15:34:29,303 [main] INFO org.eclipse.jetty.server.Server - jetty-9.0.z-SNAPSHOTWrappedArray() 2016-04-13 15:34:29,336 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - Started WrappedArray(o.e.j.s.ServletContextHandler@3aefe5e5{/,null,AVAILA 2016-04-13 15:34:29,351 [main] INFO org.eclipse.jetty.server.ServerConnector - Started WrappedArray(ServerConnector@71b1176b{HTTP/1.1}{0.0.0.0:7000}) 2016-04-13 15:34:29,352 [main] INFO ly.stealth.mesos.kafka.HttpServer$ - started on port 7000 I0413 15:34:29.420655 12601 sched.cpp:222] Version: 0.27.1 I0413 15:34:29.423641 12647 sched.cpp:326] New master detected at [email protected]:5050 I0413 15:34:29.424010 12647 sched.cpp:336] No credentials provided. Attempting to register without authentication —
In a working setup, the next line logged is the UUID of the framework registered, so this is why I tcpdumped the traffic to the mesos master in my prev message. From: Justin Ryan <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, April 18, 2016 at 1:35 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: kafka-mesos still refusing to launch brokers on one cluster They don’t say anything. Mesos never attempts to start the jobs, as far as I can tell. The framework fails to register, for reasons I can’t determine, nothing is ever logged that matches a search for ‘kafka’, and the brokers in kafka-mesos broker status show as ‘launching’ or somesuch, forever. It is quite perplexing. From: Vinod Kone <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, April 18, 2016 at 12:43 PM To: user <[email protected]<mailto:[email protected]>> Subject: Re: kafka-mesos still refusing to launch brokers on one cluster On Mon, Apr 18, 2016 at 12:32 PM, Justin Ryan <[email protected]<mailto:[email protected]>> wrote: So, test is working again, but prod is still and has consistently been dead What do the master/agent/scheduler logs say regarding kafka tasks? ________________________________ P Please consider the environment before printing this e-mail The information in this electronic mail message is the sender's confidential business and may be legally privileged. It is intended solely for the addressee(s). Access to this internet electronic mail message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. The sender believes that this E-mail and any attachments were free of any virus, worm, Trojan horse, and/or malicious code when sent. This message and its attachments could have been infected during transmission. By reading the message and opening any attachments, the recipient accepts full responsibility for taking protective and remedial action about viruses and other defects. The sender's employer is not liable for any loss or damage arising in any way.
