I got it all working. Looking in the logs on the master I could see the docker images attempting to register itself, however, it seemed like the request didn't correctly go back to the container. I could see that the return IP address was the one internal to the container itself and the packets were not routing back correctly. Rather than fight docker on that one I decided to take that information and fire up the framework again. I discovered the framework itself didn't log anything in the master's logs. I dug a bit deeper and I had used 'master1' to refer to my master in the config files which resolves fine from my other boxes, however running the framework on the master itself it resolved to 127.0.1.1. I'm not sure if it's because how Mesos binds to it's IP or if because of our pretty tight firewall rules, but it was blocked from registering correctly. Once I switched the properties file to using the open, public IP of the master everything fired right up. This was useful Mesos debugging experience.
Thanks for the nudge that lead to resolution :) Will On Thu, Jun 4, 2015 at 6:55 PM, Joe Stein <[email protected]> wrote: > logs.dir is a kafka broker configuration you can override for where the > broker data is stored (in case you want to write it outside the sandbox). > > If you don't have any active/failed/killed/staged broker tasks then you > should check the mesos logs to see why maybe offers are not going out, you > should be seeing offers in the kafka scheduler logs e.g. > > 2015-06-04 22:53:29,641 [Thread-16] INFO > ly.stealth.mesos.kafka.Scheduler$ - [resourceOffers] > > slave1#O1966 cpus:1.70 mem:1792.00 disk:35164.00 > ports:[31839..32000][31000..31630][31633..31774][31778..31834] > > slave2#O1967 cpus:1.80 mem:2304.00 disk:35164.00 > ports:[31000..31230][31235..31597][31600..32000] > > slave3#O1968 cpus:1.60 mem:1536.00 disk:35164.00 > ports:[31045..31056][31061..31103][31000..31042][31105..31589][31592..32000] > > ~ Joe Stein > - - - - - - - - - - - - - - - - - > > http://www.stealth.ly > - - - - - - - - - - - - - - - - - > > On Thu, Jun 4, 2015 at 6:50 PM, William Francis <[email protected]> > wrote: > >> >> Hi Joe - >> >> Everything is the latest version as checked out from github today. At >> this point I'm just following the directions in the README.md to see if I >> can get it working. >> >> I see the scheduler registers itself as a framework on the mesos master: >> >> IDHostUserNameActive TasksCPUsMemMax ShareRegisteredRe-Registered >> …5050-25434-0004 >> <http://wfrancis-centos7-master1:5050/#/frameworks/20150603-233921-184621996-5050-25434-0004> >> 2440709ddd51rootkafka000 B0%an hour ago- >> but I don't see anything else launched. >> >> However, I did update my broker 0 to more memory/heap in the hopes that >> maybe that would rectify the issue, but it did not: >> >> [root@wfrancis-centos7-master1 kafka]# ./kafka-mesos.sh update 0 --heap >> 512 --mem 1024 --api=http://slave2:7000 >> Broker updated >> >> broker: >> id: 0 >> active: false >> state: stopped >> resources: cpus:0.50, mem:1024, heap:512 >> options: log.dirs=/tmp/log >> failover: delay:10s, max-delay:60s >> >> [root@wfrancis-centos7-master1 kafka]# ./kafka-mesos.sh start 0 --api= >> http://slave2:7000 >> Error: Broker 0 scheduled to start. Got timeout >> >> I also set a log.dirs in the hopes of getting some useful output, but >> don't see any in the container or outside of it. >> >> other ideas welcome! thanks. >> >> >> >> On Thu, Jun 4, 2015 at 5:54 PM, Joe Stein <[email protected]> wrote: >> >>> Hi William, do you have staging/running/failed/killed broker tasks? >>> First thought looking at the status the issue could be that you have the >>> heap == mem, when setting it so small for testing usually twice the size >>> (assuming you have cgroups for your isolator). What version are you using? >>> >>> ~ Joe Stein >>> - - - - - - - - - - - - - - - - - >>> >>> http://www.stealth.ly >>> - - - - - - - - - - - - - - - - - >>> >>> On Thu, Jun 4, 2015 at 5:38 PM, William Francis <[email protected]> >>> wrote: >>> >>>> >>>> Hello - I have Mesos and Marathon running on centos 7. I'm trying to >>>> get Kafka running but am having difficulty starting the brokers. I can >>>> launch the scheduler directly as a framework or as a docker container and >>>> that is fine (I think) but when I try to launch the brokers they time out >>>> and I'm not aware of any logs I can look through to find out what's going >>>> on. >>>> >>>> In this example I start up a kafa-mesos docker container on my slave2 >>>> and then attempt to control it from master1. >>>> >>>> This is all with java 1.7.0 >>>> >>>> docker run -it -p 7000:7000 --add-host=master1:172.27.1.11 >>>> root/kafka-mesos ./kafka-mesos.sh scheduler --master=master1:5050 >>>> --zk=master1:2181 --api=http://172.27.2.12:7000 >>>> --storage=file:kafka-mesos.json >>>> >>>> 2015-06-04 20:46:04,664 [main] INFO ly.stealth.mesos.kafka.Scheduler$ >>>> - Starting Scheduler$: >>>> debug: false, storage: file:kafka-mesos.json >>>> mesos: master=master1:5050, user=<default>, principal=<none>, >>>> secret=<none> >>>> framework: name=kafka, role=*, timeout=30d >>>> api: http://172.27.2.12:7000, zk: master1:2181, jre: <none> >>>> 2015-06-04 20:46:04,780 [main] INFO org.eclipse.jetty.server.Server - >>>> jetty-9.0.z-SNAPSHOTWrappedArray() >>>> 2015-06-04 20:46:04,821 [main] INFO >>>> org.eclipse.jetty.server.handler.ContextHandler - Started >>>> WrappedArray(o.e.j.s.ServletContextHandler@13a15f06{/,null,AVAILABLE}) >>>> 2015-06-04 20:46:04,850 [main] INFO >>>> org.eclipse.jetty.server.ServerConnector - Started >>>> WrappedArray(ServerConnector@2f37e8bc{HTTP/1.1}{0.0.0.0:7000}) >>>> 2015-06-04 20:46:04,851 [main] INFO ly.stealth.mesos.kafka.HttpServer$ >>>> - started on port 7000 >>>> I0604 20:46:04.943804 16 sched.cpp:157] Version: 0.22.1 >>>> I0604 20:46:04.947131 43 sched.cpp:254] New master detected at >>>> [email protected]:5050 >>>> I0604 20:46:04.947571 43 sched.cpp:264] No credentials provided. >>>> Attempting to register without authentication >>>> >>>> then back on my master1: >>>> >>>> [root@wfrancis-centos7-master1 kafka]# ./kafka-mesos.sh status --api= >>>> http://slave2:7000 >>>> Cluster status received >>>> >>>> cluster: >>>> brokers: >>>> >>>> [root@wfrancis-centos7-master1 kafka]# ./kafka-mesos.sh add 0 --api= >>>> http://slave2:7000 >>>> Broker added >>>> >>>> broker: >>>> id: 0 >>>> active: false >>>> state: stopped >>>> resources: cpus:0.50, mem:128, heap:128 >>>> failover: delay:10s, max-delay:60s >>>> >>>> [root@wfrancis-centos7-master1 kafka]# ./kafka-mesos.sh start 0 --api= >>>> http://slave2:7000 >>>> Error: Broker 0 scheduled to start. Got timeout >>>> >>>> [root@wfrancis-centos7-master1 kafka]# ./kafka-mesos.sh status --api= >>>> http://slave2:7000 >>>> Cluster status received >>>> >>>> cluster: >>>> brokers: >>>> id: 0 >>>> active: true >>>> state: starting >>>> resources: cpus:0.50, mem:128, heap:128 >>>> failover: delay:10s, max-delay:60s >>>> >>>> However the broker never starts. Ideas of how to debug this? >>>> >>>> >>>> >>> >> >
