Hi William, If you wanted to continue to run the kafka framework in a docker container you will need to run the docker container with `--net=host` so that that framework will register from the host.
The fact that it register with `127.0.1.1` is probably beacuse /etc/hosts on the master is configured to have it's loopback interface assigned to the machine name, if this is removed from /etc/hosts it should delegate back to the default interface. --Ben On Thu, Jun 4, 2015 at 5:40 PM, William Francis <[email protected]> wrote: > > I got it all working. Looking in the logs on the master I could see the > docker images attempting to register itself, however, it seemed like the > request didn't correctly go back to the container. I could see that the > return IP address was the one internal to the container itself and the > packets were not routing back correctly. Rather than fight docker on that > one I decided to take that information and fire up the framework again. I > discovered the framework itself didn't log anything in the master's logs. I > dug a bit deeper and I had used 'master1' to refer to my master in the > config files which resolves fine from my other boxes, however running the > framework on the master itself it resolved to 127.0.1.1. I'm not sure if > it's because how Mesos binds to it's IP or if because of our pretty tight > firewall rules, but it was blocked from registering correctly. Once I > switched the properties file to using the open, public IP of the master > everything fired right up. This was useful Mesos debugging experience. > > Thanks for the nudge that lead to resolution :) > > Will > > > On Thu, Jun 4, 2015 at 6:55 PM, Joe Stein <[email protected]> wrote: > >> logs.dir is a kafka broker configuration you can override for where the >> broker data is stored (in case you want to write it outside the sandbox). >> >> If you don't have any active/failed/killed/staged broker tasks then you >> should check the mesos logs to see why maybe offers are not going out, you >> should be seeing offers in the kafka scheduler logs e.g. >> >> 2015-06-04 22:53:29,641 [Thread-16] INFO >> ly.stealth.mesos.kafka.Scheduler$ - [resourceOffers] >> >> slave1#O1966 cpus:1.70 mem:1792.00 disk:35164.00 >> ports:[31839..32000][31000..31630][31633..31774][31778..31834] >> >> slave2#O1967 cpus:1.80 mem:2304.00 disk:35164.00 >> ports:[31000..31230][31235..31597][31600..32000] >> >> slave3#O1968 cpus:1.60 mem:1536.00 disk:35164.00 >> ports:[31045..31056][31061..31103][31000..31042][31105..31589][31592..32000] >> >> ~ Joe Stein >> - - - - - - - - - - - - - - - - - >> >> http://www.stealth.ly >> - - - - - - - - - - - - - - - - - >> >> On Thu, Jun 4, 2015 at 6:50 PM, William Francis <[email protected]> >> wrote: >> >>> >>> Hi Joe - >>> >>> Everything is the latest version as checked out from github today. At >>> this point I'm just following the directions in the README.md to see if I >>> can get it working. >>> >>> I see the scheduler registers itself as a framework on the mesos master: >>> >>> IDHostUserNameActive TasksCPUsMemMax ShareRegisteredRe-Registered >>> …5050-25434-0004 >>> <http://wfrancis-centos7-master1:5050/#/frameworks/20150603-233921-184621996-5050-25434-0004> >>> 2440709ddd51rootkafka000 B0%an hour ago- >>> but I don't see anything else launched. >>> >>> However, I did update my broker 0 to more memory/heap in the hopes that >>> maybe that would rectify the issue, but it did not: >>> >>> [root@wfrancis-centos7-master1 kafka]# ./kafka-mesos.sh update 0 --heap >>> 512 --mem 1024 --api=http://slave2:7000 >>> Broker updated >>> >>> broker: >>> id: 0 >>> active: false >>> state: stopped >>> resources: cpus:0.50, mem:1024, heap:512 >>> options: log.dirs=/tmp/log >>> failover: delay:10s, max-delay:60s >>> >>> [root@wfrancis-centos7-master1 kafka]# ./kafka-mesos.sh start 0 --api= >>> http://slave2:7000 >>> Error: Broker 0 scheduled to start. Got timeout >>> >>> I also set a log.dirs in the hopes of getting some useful output, but >>> don't see any in the container or outside of it. >>> >>> other ideas welcome! thanks. >>> >>> >>> >>> On Thu, Jun 4, 2015 at 5:54 PM, Joe Stein <[email protected]> wrote: >>> >>>> Hi William, do you have staging/running/failed/killed broker tasks? >>>> First thought looking at the status the issue could be that you have the >>>> heap == mem, when setting it so small for testing usually twice the size >>>> (assuming you have cgroups for your isolator). What version are you using? >>>> >>>> ~ Joe Stein >>>> - - - - - - - - - - - - - - - - - >>>> >>>> http://www.stealth.ly >>>> - - - - - - - - - - - - - - - - - >>>> >>>> On Thu, Jun 4, 2015 at 5:38 PM, William Francis <[email protected]> >>>> wrote: >>>> >>>>> >>>>> Hello - I have Mesos and Marathon running on centos 7. I'm trying to >>>>> get Kafka running but am having difficulty starting the brokers. I can >>>>> launch the scheduler directly as a framework or as a docker container and >>>>> that is fine (I think) but when I try to launch the brokers they time out >>>>> and I'm not aware of any logs I can look through to find out what's going >>>>> on. >>>>> >>>>> In this example I start up a kafa-mesos docker container on my slave2 >>>>> and then attempt to control it from master1. >>>>> >>>>> This is all with java 1.7.0 >>>>> >>>>> docker run -it -p 7000:7000 --add-host=master1:172.27.1.11 >>>>> root/kafka-mesos ./kafka-mesos.sh scheduler --master=master1:5050 >>>>> --zk=master1:2181 --api=http://172.27.2.12:7000 >>>>> --storage=file:kafka-mesos.json >>>>> >>>>> 2015-06-04 20:46:04,664 [main] INFO ly.stealth.mesos.kafka.Scheduler$ >>>>> - Starting Scheduler$: >>>>> debug: false, storage: file:kafka-mesos.json >>>>> mesos: master=master1:5050, user=<default>, principal=<none>, >>>>> secret=<none> >>>>> framework: name=kafka, role=*, timeout=30d >>>>> api: http://172.27.2.12:7000, zk: master1:2181, jre: <none> >>>>> 2015-06-04 20:46:04,780 [main] INFO org.eclipse.jetty.server.Server >>>>> - jetty-9.0.z-SNAPSHOTWrappedArray() >>>>> 2015-06-04 20:46:04,821 [main] INFO >>>>> org.eclipse.jetty.server.handler.ContextHandler - Started >>>>> WrappedArray(o.e.j.s.ServletContextHandler@13a15f06{/,null,AVAILABLE}) >>>>> 2015-06-04 20:46:04,850 [main] INFO >>>>> org.eclipse.jetty.server.ServerConnector - Started >>>>> WrappedArray(ServerConnector@2f37e8bc{HTTP/1.1}{0.0.0.0:7000}) >>>>> 2015-06-04 20:46:04,851 [main] INFO >>>>> ly.stealth.mesos.kafka.HttpServer$ - started on port 7000 >>>>> I0604 20:46:04.943804 16 sched.cpp:157] Version: 0.22.1 >>>>> I0604 20:46:04.947131 43 sched.cpp:254] New master detected at >>>>> [email protected]:5050 >>>>> I0604 20:46:04.947571 43 sched.cpp:264] No credentials provided. >>>>> Attempting to register without authentication >>>>> >>>>> then back on my master1: >>>>> >>>>> [root@wfrancis-centos7-master1 kafka]# ./kafka-mesos.sh status --api= >>>>> http://slave2:7000 >>>>> Cluster status received >>>>> >>>>> cluster: >>>>> brokers: >>>>> >>>>> [root@wfrancis-centos7-master1 kafka]# ./kafka-mesos.sh add 0 --api= >>>>> http://slave2:7000 >>>>> Broker added >>>>> >>>>> broker: >>>>> id: 0 >>>>> active: false >>>>> state: stopped >>>>> resources: cpus:0.50, mem:128, heap:128 >>>>> failover: delay:10s, max-delay:60s >>>>> >>>>> [root@wfrancis-centos7-master1 kafka]# ./kafka-mesos.sh start 0 --api= >>>>> http://slave2:7000 >>>>> Error: Broker 0 scheduled to start. Got timeout >>>>> >>>>> [root@wfrancis-centos7-master1 kafka]# ./kafka-mesos.sh status --api= >>>>> http://slave2:7000 >>>>> Cluster status received >>>>> >>>>> cluster: >>>>> brokers: >>>>> id: 0 >>>>> active: true >>>>> state: starting >>>>> resources: cpus:0.50, mem:128, heap:128 >>>>> failover: delay:10s, max-delay:60s >>>>> >>>>> However the broker never starts. Ideas of how to debug this? >>>>> >>>>> >>>>> >>>> >>> >> >
