Re: mesos/kafka issues (reviving old thread)

To close the loop on this one : With Steve Niemitz's help, I resolved the issue, kafka-scheduler was going to wrong master and the slave it was going to returned a /master/status.json which did not have any information about the registered slaves. "slaves":[ ], What did I do? 1.

Re: mesos/kafka issues (reviving old thread)

+1 to setting --debug also make sure you set the --api via CLI or properties file correctly suspectfully sounds like the issue that keeps going back and forth (or since it isn't set right not going back and forth ... ) its also why it works on master because your property file may have api

Re: mesos/kafka issues (reviving old thread)

Have you tried running the Kafka scheduler in debug? (Pass --debug to iirc). That gives you a good amount of output in stdout/stderr. Also make sure your mesos lib that the scheduler is running matches your master version. Finally, make sure the master can communicate BACK to the scheduler on

Re: mesos/kafka issues (reviving old thread)

I am running into an issue where kafka framework can't register with Mesos. In packet capture I see the POST request to subscribe on mesos master box but mesos master does not respond, neither it logs that it received the request which it usually does on any attempt by a framework. Kafka-framework

Re: mesos/kafka issues (reviving old thread)

No issues here, we've been running two 8 broker clusters for ~a month without incident, and I plan on rolling it out to one of our larger (~40 broker) clusters next week. My experience with it has been really positive so far, it just pretty much worked out of the box. I'm curious what issues

Re: mesos/kafka issues (reviving old thread)

​Hi Steve, ​ How long has it been running without problems? I have read on mailing list some people complaining that brokers sometimes disappear etc. Have you come across any such problems? Any other issues that you had to take care of? I tried to use the version you specified and also took the

Re: mesos/kafka issues (reviving old thread)

Just to chime in, I've been running the 0.9.5.1 scheduler as a task on another slave without issues. (Aurora runs the kafka-mesos scheduler in my case). On Thu, Jun 9, 2016 at 2:50 PM, Vinit Mahedia wrote: > Justin, > > When you say "working" - does it mean

Re: mesos/kafka issues (reviving old thread)

Justin, When you say "working" - does it mean kafka-scheduler still has to be on the same box as mesos-master? or you have it working without that constraint? On Wed, Jun 8, 2016 at 6:07 PM, Justin Ryan wrote: > inline > > On 6/8/16, 4:06 PM, "Justin Ryan"

Re: mesos/kafka issues (reviving old thread)

inline On 6/8/16, 4:06 PM, "Justin Ryan" wrote: >FYI, when I updated to the latest kafka-mesos (0.5.1.0) this problem went >away. FWIW, I’m actually using a branch which updates kafka to 0.10.0.0 as >well: > Correction: 0.9.5.1 (current git master) > PR for kafka

Re: mesos/kafka issues (reviving old thread)

FYI, when I updated to the latest kafka-mesos (0.5.1.0) this problem went away. FWIW, I’m actually using a branch which updates kafka to 0.10.0.0 as well: PR for kafka 0.10.0.0 (tests still fail, someone else did the bulk of porting but didn’t PR it) : https://github.com/mesos/kafka/pull/220

Re: mesos/kafka issues (reviving old thread)

Vinod, In my case, have two vagrant VMs, one where mesos-master runs and frameworks on another. This is my dev/debug setup where both marathon and kafka frameworks are failing not just kafka. But we do have a real (bare metal machines cluster) running marathon successfully but kafka framework

Re: mesos/kafka issues (reviving old thread)

Right, mentioning ZK may have been a distraction. My point was that the inactive mesos-master on the host running kafka-mesos scheduler is communicating with the active mesos-master. To elaborate, this problem currently arises when a mesos-master failover occurs. If I shut off the active

Re: mesos/kafka issues (reviving old thread)

ZK communication is little different from framework communication. In the former case, the ZK client library inside master opens up a connection to the ZK server ensemble. In the latter case, the scheduler driver inside the framework scheduler opens up a connection to the mesos master. But it is

Re: mesos/kafka issues (reviving old thread)

inline On 6/6/16, 8:40 AM, "Vinod Kone" wrote: >Have you tried running a different framework than the kafka one (maybe >marathon? or chronos?) to rule out framework related issues? I'm surprised >that it works when the scheduler and master are on the same host but not

Re: mesos/kafka issues (reviving old thread)

Hiya Vinit, I’ve made some progress, I have a conditionally working setup, and another setup which was working now failing in new ways. It does sound like your captures are similar to mine, what I found is that if I run the scheduler on, say, zk01 (which is also a mesos-master), while it is

Re: mesos/kafka issues (reviving old thread)

inline On 6/3/16, 2:19 PM, "Vinit Mahedia" wrote: >Justin, >Yeah - as long as everything is on a single box (mesos-kafka scheduler, >mesos-master, zk etc.) >things work just fine, which is what I meant by local setup. > > >I did a local cluster setup as well, 3 vagrant

Re: mesos/kafka issues (reviving old thread)

Have you tried running a different framework than the kafka one (maybe marathon? or chronos?) to rule out framework related issues? I'm surprised that it works when the scheduler and master are on the same host but not when they are different. Looks like the request packets are getting dropped

Re: mesos/kafka issues (reviving old thread)

Justin, There's certainly a bug somewhere, either in mesos kafka framework or mesos itself. If I can get this mesos master running under debugger, this would be over in maybe few hours or few days. Kafka is a stateful service so can't run on marathon directly, there has to be some

Re: mesos/kafka issues (reviving old thread)

Justin, Yeah - as long as everything is on a single box (mesos-kafka scheduler, mesos-master, zk etc.) things work just fine, which is what I meant by local setup. I did a local cluster setup as well, 3 vagrant machines, where it does *not* work. So it does not work at all if you have multi node

mesos/kafka issues (reviving old thread)

​​Hey Justin, I am running in the same issues as you mentioned in this old thread , did you resolve it? I see that kafka framework sends a POST request to register itself