Re: mesos/kafka issues (reviving old thread)

[Vinit Mahedia]

To close the loop on this one :

With Steve Niemitz's help, I resolved the issue, kafka-scheduler was going
to wrong master and the slave it was going to returned a
/master/status.json which did not have any information about the registered
slaves.

"slaves":[

   ],

What did I do?

   1. Restarted zookeeper - running the docket image on marathon.
   2. Edited kafka-mesos.properties with same content but changing seq.

It started working and I just tested it by producing one test message and
consuming it.

Thanks to all of you, who helped, for your time.

Vinit.


PS: for some reason, Steve was not CCed on one of the replies, so I worked
with him 1-1 below is the conversation thread, if anybody else runs into
the same problem.


On Mon, Jun 27, 2016 at 1:35 PM, Vinit Mahedia 
wrote:

> That exactly was the problem, it was going to the wrong master, I
> restarted zookeeper and edited the kafka-mesos.properies with the same
> content just changed the sequence of arguments, it discovered new master
> after this.
>
> Although I don't understand why it was discovering wrong master though on
> the start up, maybe stale state from ZK, but should not it get the correct
> master on startup?
>
> This is a huge help, truly appreciate.
>
>
>
> On Mon, Jun 27, 2016 at 12:44 PM, Steve Niemitz 
> wrote:
>
>> I think I know what your problem is, I've run into it before.  If you
>> don't have any slaves registered, mesos will just ignore any attempt to
>> register a framework.
>>
>> I looked at the results from /master/state.json in your packet capture
>> and you don't have any registered slaves.
>>
>> Try bringing up a slave and see if it works then.
>>
>> On Mon, Jun 27, 2016 at 3:25 PM, Vinit Mahedia 
>> wrote:
>>
>>> I replied to the thread and realized that for some reason, Joe Stein's
>>> reply did not have you the CC list, so forwarding you this.
>>>
>>>
>>> Hi Steve,
>>>
>>> Finally, I am getting the Jetty error for bad HTTP message - logs
>>> .
>>> Although I get this after a long time. Here is the packet capture.
>>>
>>> Kafka running on 10.10.17.41 and mesos-master is running on 10.10.17.68.
>>>
>>> Here is the screenshot, if you quickly want to look at the logs.
>>>
>>> Scheduler-logS
>>> ×
>>>
>>>
>>>
>>> Packet Capture with only POST showing
>>> Following the TCP stream of above capture.
>>>
>>>
>>>
>>>
>>> On Fri, Jun 24, 2016 at 6:21 AM, Joe Stein  wrote:
>>>
 +1 to setting --debug

 also make sure you set the --api via CLI or properties file correctly
 suspectfully sounds like the issue that keeps going back and forth (or
 since it isn't set right not going back and forth ... )  its also why
 it works on master because your property file may have api as localhost  or
 such...

 thinking out loud quick here but overall definitely a first thing to
 check

 regards,


 ~ Joe Stein

 On Fri, Jun 24, 2016 at 12:05 AM, Steve Niemitz 
 wrote:

> Have you tried running the Kafka scheduler in debug? (Pass --debug to
> iirc). That gives you a good amount of output in stdout/stderr.
>
> Also make sure your mesos lib that the scheduler is running matches
> your
> master version.
>
> Finally, make sure the master can communicate BACK to the scheduler on
> whatever port you set as the LIBPROCESS_PORT on the scheduler.
> (Firewall
> rules and such)
>
> If you want to post the stderr/stdout logs from the scheduler (with
> debug
> on) I can take a look.
> On Jun 23, 2016 4:41 PM, "Vinit Mahedia" 
> wrote:
>
> > I am running into an issue where kafka framework can't register with
> > Mesos. In packet capture I see the POST request to subscribe on mesos
> > master box but mesos master does not respond, neither it logs that it
> > received the request which it usually does on any attempt by a
> framework.
> > Kafka-framework keeps re-sending the request and eventually gives up.
> >
> > running locally, Mesos in VM1 and kafka-framework on VM2 works fine,
> also
> > some people here reported that putting kafka-framework on mesos
> master
> > works as well, but that is not actually working deployment.
> >
> > I can provide more information. Thanks for offering help, I really
> > appreciate your time.
> >
> > On Fri, Jun 17, 2016 at 1:36 PM, Steve Niemitz  >
> > wrote:
> >
> >> No issues here, we've been running two 8 broker clusters for ~a
> month
> >> without incident, and I plan on rolling it out to one of our larger
> (~40
> >> broker) clusters next week.
> >>
> >> My experience with it has been really positive so far, it just
> pretty
> 

Re: mesos/kafka issues (reviving old thread)

[Joe Stein]

+1 to setting --debug

also make sure you set the --api via CLI or properties file correctly
suspectfully sounds like the issue that keeps going back and forth (or
since it isn't set right not going back and forth ... )  its also why
it works on master because your property file may have api as localhost  or
such...

thinking out loud quick here but overall definitely a first thing to check

regards,


~ Joe Stein

On Fri, Jun 24, 2016 at 12:05 AM, Steve Niemitz  wrote:

> Have you tried running the Kafka scheduler in debug? (Pass --debug to
> iirc). That gives you a good amount of output in stdout/stderr.
>
> Also make sure your mesos lib that the scheduler is running matches your
> master version.
>
> Finally, make sure the master can communicate BACK to the scheduler on
> whatever port you set as the LIBPROCESS_PORT on the scheduler. (Firewall
> rules and such)
>
> If you want to post the stderr/stdout logs from the scheduler (with debug
> on) I can take a look.
> On Jun 23, 2016 4:41 PM, "Vinit Mahedia"  wrote:
>
> > I am running into an issue where kafka framework can't register with
> > Mesos. In packet capture I see the POST request to subscribe on mesos
> > master box but mesos master does not respond, neither it logs that it
> > received the request which it usually does on any attempt by a framework.
> > Kafka-framework keeps re-sending the request and eventually gives up.
> >
> > running locally, Mesos in VM1 and kafka-framework on VM2 works fine, also
> > some people here reported that putting kafka-framework on mesos master
> > works as well, but that is not actually working deployment.
> >
> > I can provide more information. Thanks for offering help, I really
> > appreciate your time.
> >
> > On Fri, Jun 17, 2016 at 1:36 PM, Steve Niemitz 
> > wrote:
> >
> >> No issues here, we've been running two 8 broker clusters for ~a month
> >> without incident, and I plan on rolling it out to one of our larger (~40
> >> broker) clusters next week.
> >>
> >> My experience with it has been really positive so far, it just pretty
> >> much worked out of the box.   I'm curious what issues you ran into,
> happy
> >> to try to help if you want!
> >>
> >> On Fri, Jun 17, 2016 at 3:31 PM, Vinit Mahedia 
> >> wrote:
> >>
> >>> ​Hi Steve, ​
> >>>
> >>> How long has it been running without problems? I have read on mailing
> >>> list
> >>> some people complaining that brokers sometimes disappear etc. Have you
> >>> come
> >>> across any such problems? Any other issues that you had to take care
> of?
> >>>
> >>> I tried to use the version you specified and also took the latest
> release
> >>> of kafka without luck although I am happy to have found someone for
> whom
> >>> it's working.
> >>>
> >>>
> >>> On Fri, Jun 10, 2016 at 3:57 PM, Steve Niemitz 
> >>> wrote:
> >>>
> >>> > Just to chime in, I've been running the 0.9.5.1 scheduler as a task
> on
> >>> > another slave without issues. (Aurora runs the kafka-mesos scheduler
> >>> in my
> >>> > case).
> >>> >
> >>> > On Thu, Jun 9, 2016 at 2:50 PM, Vinit Mahedia <
> vinitmahe...@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Justin,
> >>> > >
> >>> > > When you say "working" - does it mean kafka-scheduler still has to
> >>> be on
> >>> > > the same box as mesos-master? or you
> >>> > > have it working without that constraint?
> >>> > >
> >>> > > On Wed, Jun 8, 2016 at 6:07 PM, Justin Ryan 
> >>> > wrote:
> >>> > >
> >>> > > > inline
> >>> > > >
> >>> > > > On 6/8/16, 4:06 PM, "Justin Ryan"  wrote:
> >>> > > >
> >>> > > > >FYI, when I updated to the latest kafka-mesos (0.5.1.0) this
> >>> problem
> >>> > > went
> >>> > > > away.  FWIW, I’m actually using a branch which updates kafka to
> >>> > 0.10.0.0
> >>> > > as
> >>> > > > well:
> >>> > > > >
> >>> > > >
> >>> > > > Correction: 0.9.5.1 (current git master)
> >>> > > >
> >>> > > > > PR for kafka 0.10.0.0 (tests still fail, someone else did the
> >>> bulk of
> >>> > > > porting but didn’t PR it) :
> >>> https://github.com/mesos/kafka/pull/220
> >>> > > > >   ( ./gradlew jar –x test gets a successful build )
> >>> > > > >
> >>> > > > > Issue for the problem discussed in this thread:
> >>> > > > https://github.com/mesos/kafka/issues/199
> >>> > > > >
> >>> > > > >Cheers!
> >>> > > > >
> >>> > > >
> >>> > > >
> >>> > > > 
> >>> > > >
> >>> > > > P Please consider the environment before printing this e-mail
> >>> > > >
> >>> > > > The information in this electronic mail message is the sender's
> >>> > > > confidential business and may be legally privileged. It is
> intended
> >>> > > solely
> >>> > > > for the addressee(s). Access to this internet electronic mail
> >>> message
> >>> > by
> >>> > > > anyone else is unauthorized. If you are not the intended
> >>> recipient, any
> >>> > > > disclosure, copying, distribution or any action 

Re: mesos/kafka issues (reviving old thread)

[Steve Niemitz]

Have you tried running the Kafka scheduler in debug? (Pass --debug to
iirc). That gives you a good amount of output in stdout/stderr.

Also make sure your mesos lib that the scheduler is running matches your
master version.

Finally, make sure the master can communicate BACK to the scheduler on
whatever port you set as the LIBPROCESS_PORT on the scheduler. (Firewall
rules and such)

If you want to post the stderr/stdout logs from the scheduler (with debug
on) I can take a look.
On Jun 23, 2016 4:41 PM, "Vinit Mahedia"  wrote:

> I am running into an issue where kafka framework can't register with
> Mesos. In packet capture I see the POST request to subscribe on mesos
> master box but mesos master does not respond, neither it logs that it
> received the request which it usually does on any attempt by a framework.
> Kafka-framework keeps re-sending the request and eventually gives up.
>
> running locally, Mesos in VM1 and kafka-framework on VM2 works fine, also
> some people here reported that putting kafka-framework on mesos master
> works as well, but that is not actually working deployment.
>
> I can provide more information. Thanks for offering help, I really
> appreciate your time.
>
> On Fri, Jun 17, 2016 at 1:36 PM, Steve Niemitz 
> wrote:
>
>> No issues here, we've been running two 8 broker clusters for ~a month
>> without incident, and I plan on rolling it out to one of our larger (~40
>> broker) clusters next week.
>>
>> My experience with it has been really positive so far, it just pretty
>> much worked out of the box.   I'm curious what issues you ran into, happy
>> to try to help if you want!
>>
>> On Fri, Jun 17, 2016 at 3:31 PM, Vinit Mahedia 
>> wrote:
>>
>>> ​Hi Steve, ​
>>>
>>> How long has it been running without problems? I have read on mailing
>>> list
>>> some people complaining that brokers sometimes disappear etc. Have you
>>> come
>>> across any such problems? Any other issues that you had to take care of?
>>>
>>> I tried to use the version you specified and also took the latest release
>>> of kafka without luck although I am happy to have found someone for whom
>>> it's working.
>>>
>>>
>>> On Fri, Jun 10, 2016 at 3:57 PM, Steve Niemitz 
>>> wrote:
>>>
>>> > Just to chime in, I've been running the 0.9.5.1 scheduler as a task on
>>> > another slave without issues. (Aurora runs the kafka-mesos scheduler
>>> in my
>>> > case).
>>> >
>>> > On Thu, Jun 9, 2016 at 2:50 PM, Vinit Mahedia 
>>> > wrote:
>>> >
>>> > > Justin,
>>> > >
>>> > > When you say "working" - does it mean kafka-scheduler still has to
>>> be on
>>> > > the same box as mesos-master? or you
>>> > > have it working without that constraint?
>>> > >
>>> > > On Wed, Jun 8, 2016 at 6:07 PM, Justin Ryan 
>>> > wrote:
>>> > >
>>> > > > inline
>>> > > >
>>> > > > On 6/8/16, 4:06 PM, "Justin Ryan"  wrote:
>>> > > >
>>> > > > >FYI, when I updated to the latest kafka-mesos (0.5.1.0) this
>>> problem
>>> > > went
>>> > > > away.  FWIW, I’m actually using a branch which updates kafka to
>>> > 0.10.0.0
>>> > > as
>>> > > > well:
>>> > > > >
>>> > > >
>>> > > > Correction: 0.9.5.1 (current git master)
>>> > > >
>>> > > > > PR for kafka 0.10.0.0 (tests still fail, someone else did the
>>> bulk of
>>> > > > porting but didn’t PR it) :
>>> https://github.com/mesos/kafka/pull/220
>>> > > > >   ( ./gradlew jar –x test gets a successful build )
>>> > > > >
>>> > > > > Issue for the problem discussed in this thread:
>>> > > > https://github.com/mesos/kafka/issues/199
>>> > > > >
>>> > > > >Cheers!
>>> > > > >
>>> > > >
>>> > > >
>>> > > > 
>>> > > >
>>> > > > P Please consider the environment before printing this e-mail
>>> > > >
>>> > > > The information in this electronic mail message is the sender's
>>> > > > confidential business and may be legally privileged. It is intended
>>> > > solely
>>> > > > for the addressee(s). Access to this internet electronic mail
>>> message
>>> > by
>>> > > > anyone else is unauthorized. If you are not the intended
>>> recipient, any
>>> > > > disclosure, copying, distribution or any action taken or omitted
>>> to be
>>> > > > taken in reliance on it is prohibited and may be unlawful. The
>>> sender
>>> > > > believes that this E-mail and any attachments were free of any
>>> virus,
>>> > > worm,
>>> > > > Trojan horse, and/or malicious code when sent. This message and its
>>> > > > attachments could have been infected during transmission. By
>>> reading
>>> > the
>>> > > > message and opening any attachments, the recipient accepts full
>>> > > > responsibility for taking protective and remedial action about
>>> viruses
>>> > > and
>>> > > > other defects. The sender's employer is not liable for any loss or
>>> > damage
>>> > > > arising in any way.
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > ~Vinit
>>> > >
>>> 

Re: mesos/kafka issues (reviving old thread)

[Vinit Mahedia]

I am running into an issue where kafka framework can't register with Mesos.
In packet capture I see the POST request to subscribe on mesos master box
but mesos master does not respond, neither it logs that it received the
request which it usually does on any attempt by a framework.
Kafka-framework keeps re-sending the request and eventually gives up.

running locally, Mesos in VM1 and kafka-framework on VM2 works fine, also
some people here reported that putting kafka-framework on mesos master
works as well, but that is not actually working deployment.

I can provide more information. Thanks for offering help, I really
appreciate your time.

On Fri, Jun 17, 2016 at 1:36 PM, Steve Niemitz  wrote:

> No issues here, we've been running two 8 broker clusters for ~a month
> without incident, and I plan on rolling it out to one of our larger (~40
> broker) clusters next week.
>
> My experience with it has been really positive so far, it just pretty much
> worked out of the box.   I'm curious what issues you ran into, happy to try
> to help if you want!
>
> On Fri, Jun 17, 2016 at 3:31 PM, Vinit Mahedia 
> wrote:
>
>> ​Hi Steve, ​
>>
>> How long has it been running without problems? I have read on mailing list
>> some people complaining that brokers sometimes disappear etc. Have you
>> come
>> across any such problems? Any other issues that you had to take care of?
>>
>> I tried to use the version you specified and also took the latest release
>> of kafka without luck although I am happy to have found someone for whom
>> it's working.
>>
>>
>> On Fri, Jun 10, 2016 at 3:57 PM, Steve Niemitz 
>> wrote:
>>
>> > Just to chime in, I've been running the 0.9.5.1 scheduler as a task on
>> > another slave without issues. (Aurora runs the kafka-mesos scheduler in
>> my
>> > case).
>> >
>> > On Thu, Jun 9, 2016 at 2:50 PM, Vinit Mahedia 
>> > wrote:
>> >
>> > > Justin,
>> > >
>> > > When you say "working" - does it mean kafka-scheduler still has to be
>> on
>> > > the same box as mesos-master? or you
>> > > have it working without that constraint?
>> > >
>> > > On Wed, Jun 8, 2016 at 6:07 PM, Justin Ryan 
>> > wrote:
>> > >
>> > > > inline
>> > > >
>> > > > On 6/8/16, 4:06 PM, "Justin Ryan"  wrote:
>> > > >
>> > > > >FYI, when I updated to the latest kafka-mesos (0.5.1.0) this
>> problem
>> > > went
>> > > > away.  FWIW, I’m actually using a branch which updates kafka to
>> > 0.10.0.0
>> > > as
>> > > > well:
>> > > > >
>> > > >
>> > > > Correction: 0.9.5.1 (current git master)
>> > > >
>> > > > > PR for kafka 0.10.0.0 (tests still fail, someone else did the
>> bulk of
>> > > > porting but didn’t PR it) : https://github.com/mesos/kafka/pull/220
>> > > > >   ( ./gradlew jar –x test gets a successful build )
>> > > > >
>> > > > > Issue for the problem discussed in this thread:
>> > > > https://github.com/mesos/kafka/issues/199
>> > > > >
>> > > > >Cheers!
>> > > > >
>> > > >
>> > > >
>> > > > 
>> > > >
>> > > > P Please consider the environment before printing this e-mail
>> > > >
>> > > > The information in this electronic mail message is the sender's
>> > > > confidential business and may be legally privileged. It is intended
>> > > solely
>> > > > for the addressee(s). Access to this internet electronic mail
>> message
>> > by
>> > > > anyone else is unauthorized. If you are not the intended recipient,
>> any
>> > > > disclosure, copying, distribution or any action taken or omitted to
>> be
>> > > > taken in reliance on it is prohibited and may be unlawful. The
>> sender
>> > > > believes that this E-mail and any attachments were free of any
>> virus,
>> > > worm,
>> > > > Trojan horse, and/or malicious code when sent. This message and its
>> > > > attachments could have been infected during transmission. By reading
>> > the
>> > > > message and opening any attachments, the recipient accepts full
>> > > > responsibility for taking protective and remedial action about
>> viruses
>> > > and
>> > > > other defects. The sender's employer is not liable for any loss or
>> > damage
>> > > > arising in any way.
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > ~Vinit
>> > >
>> >
>>
>>
>>
>> --
>> ~Vinit
>>
>
>


-- 
~Vinit

Re: mesos/kafka issues (reviving old thread)

[Steve Niemitz]

No issues here, we've been running two 8 broker clusters for ~a month
without incident, and I plan on rolling it out to one of our larger (~40
broker) clusters next week.

My experience with it has been really positive so far, it just pretty much
worked out of the box.   I'm curious what issues you ran into, happy to try
to help if you want!

On Fri, Jun 17, 2016 at 3:31 PM, Vinit Mahedia 
wrote:

> ​Hi Steve, ​
>
> How long has it been running without problems? I have read on mailing list
> some people complaining that brokers sometimes disappear etc. Have you come
> across any such problems? Any other issues that you had to take care of?
>
> I tried to use the version you specified and also took the latest release
> of kafka without luck although I am happy to have found someone for whom
> it's working.
>
>
> On Fri, Jun 10, 2016 at 3:57 PM, Steve Niemitz 
> wrote:
>
> > Just to chime in, I've been running the 0.9.5.1 scheduler as a task on
> > another slave without issues. (Aurora runs the kafka-mesos scheduler in
> my
> > case).
> >
> > On Thu, Jun 9, 2016 at 2:50 PM, Vinit Mahedia 
> > wrote:
> >
> > > Justin,
> > >
> > > When you say "working" - does it mean kafka-scheduler still has to be
> on
> > > the same box as mesos-master? or you
> > > have it working without that constraint?
> > >
> > > On Wed, Jun 8, 2016 at 6:07 PM, Justin Ryan 
> > wrote:
> > >
> > > > inline
> > > >
> > > > On 6/8/16, 4:06 PM, "Justin Ryan"  wrote:
> > > >
> > > > >FYI, when I updated to the latest kafka-mesos (0.5.1.0) this problem
> > > went
> > > > away.  FWIW, I’m actually using a branch which updates kafka to
> > 0.10.0.0
> > > as
> > > > well:
> > > > >
> > > >
> > > > Correction: 0.9.5.1 (current git master)
> > > >
> > > > > PR for kafka 0.10.0.0 (tests still fail, someone else did the bulk
> of
> > > > porting but didn’t PR it) : https://github.com/mesos/kafka/pull/220
> > > > >   ( ./gradlew jar –x test gets a successful build )
> > > > >
> > > > > Issue for the problem discussed in this thread:
> > > > https://github.com/mesos/kafka/issues/199
> > > > >
> > > > >Cheers!
> > > > >
> > > >
> > > >
> > > > 
> > > >
> > > > P Please consider the environment before printing this e-mail
> > > >
> > > > The information in this electronic mail message is the sender's
> > > > confidential business and may be legally privileged. It is intended
> > > solely
> > > > for the addressee(s). Access to this internet electronic mail message
> > by
> > > > anyone else is unauthorized. If you are not the intended recipient,
> any
> > > > disclosure, copying, distribution or any action taken or omitted to
> be
> > > > taken in reliance on it is prohibited and may be unlawful. The sender
> > > > believes that this E-mail and any attachments were free of any virus,
> > > worm,
> > > > Trojan horse, and/or malicious code when sent. This message and its
> > > > attachments could have been infected during transmission. By reading
> > the
> > > > message and opening any attachments, the recipient accepts full
> > > > responsibility for taking protective and remedial action about
> viruses
> > > and
> > > > other defects. The sender's employer is not liable for any loss or
> > damage
> > > > arising in any way.
> > > >
> > >
> > >
> > >
> > > --
> > > ~Vinit
> > >
> >
>
>
>
> --
> ~Vinit
>

Re: mesos/kafka issues (reviving old thread)

[Vinit Mahedia]

​Hi Steve, ​

How long has it been running without problems? I have read on mailing list
some people complaining that brokers sometimes disappear etc. Have you come
across any such problems? Any other issues that you had to take care of?

I tried to use the version you specified and also took the latest release
of kafka without luck although I am happy to have found someone for whom
it's working.


On Fri, Jun 10, 2016 at 3:57 PM, Steve Niemitz  wrote:

> Just to chime in, I've been running the 0.9.5.1 scheduler as a task on
> another slave without issues. (Aurora runs the kafka-mesos scheduler in my
> case).
>
> On Thu, Jun 9, 2016 at 2:50 PM, Vinit Mahedia 
> wrote:
>
> > Justin,
> >
> > When you say "working" - does it mean kafka-scheduler still has to be on
> > the same box as mesos-master? or you
> > have it working without that constraint?
> >
> > On Wed, Jun 8, 2016 at 6:07 PM, Justin Ryan 
> wrote:
> >
> > > inline
> > >
> > > On 6/8/16, 4:06 PM, "Justin Ryan"  wrote:
> > >
> > > >FYI, when I updated to the latest kafka-mesos (0.5.1.0) this problem
> > went
> > > away.  FWIW, I’m actually using a branch which updates kafka to
> 0.10.0.0
> > as
> > > well:
> > > >
> > >
> > > Correction: 0.9.5.1 (current git master)
> > >
> > > > PR for kafka 0.10.0.0 (tests still fail, someone else did the bulk of
> > > porting but didn’t PR it) : https://github.com/mesos/kafka/pull/220
> > > >   ( ./gradlew jar –x test gets a successful build )
> > > >
> > > > Issue for the problem discussed in this thread:
> > > https://github.com/mesos/kafka/issues/199
> > > >
> > > >Cheers!
> > > >
> > >
> > >
> > > 
> > >
> > > P Please consider the environment before printing this e-mail
> > >
> > > The information in this electronic mail message is the sender's
> > > confidential business and may be legally privileged. It is intended
> > solely
> > > for the addressee(s). Access to this internet electronic mail message
> by
> > > anyone else is unauthorized. If you are not the intended recipient, any
> > > disclosure, copying, distribution or any action taken or omitted to be
> > > taken in reliance on it is prohibited and may be unlawful. The sender
> > > believes that this E-mail and any attachments were free of any virus,
> > worm,
> > > Trojan horse, and/or malicious code when sent. This message and its
> > > attachments could have been infected during transmission. By reading
> the
> > > message and opening any attachments, the recipient accepts full
> > > responsibility for taking protective and remedial action about viruses
> > and
> > > other defects. The sender's employer is not liable for any loss or
> damage
> > > arising in any way.
> > >
> >
> >
> >
> > --
> > ~Vinit
> >
>



-- 
~Vinit

Re: mesos/kafka issues (reviving old thread)

[Steve Niemitz]

Just to chime in, I've been running the 0.9.5.1 scheduler as a task on
another slave without issues. (Aurora runs the kafka-mesos scheduler in my
case).

On Thu, Jun 9, 2016 at 2:50 PM, Vinit Mahedia 
wrote:

> Justin,
>
> When you say "working" - does it mean kafka-scheduler still has to be on
> the same box as mesos-master? or you
> have it working without that constraint?
>
> On Wed, Jun 8, 2016 at 6:07 PM, Justin Ryan  wrote:
>
> > inline
> >
> > On 6/8/16, 4:06 PM, "Justin Ryan"  wrote:
> >
> > >FYI, when I updated to the latest kafka-mesos (0.5.1.0) this problem
> went
> > away.  FWIW, I’m actually using a branch which updates kafka to 0.10.0.0
> as
> > well:
> > >
> >
> > Correction: 0.9.5.1 (current git master)
> >
> > > PR for kafka 0.10.0.0 (tests still fail, someone else did the bulk of
> > porting but didn’t PR it) : https://github.com/mesos/kafka/pull/220
> > >   ( ./gradlew jar –x test gets a successful build )
> > >
> > > Issue for the problem discussed in this thread:
> > https://github.com/mesos/kafka/issues/199
> > >
> > >Cheers!
> > >
> >
> >
> > 
> >
> > P Please consider the environment before printing this e-mail
> >
> > The information in this electronic mail message is the sender's
> > confidential business and may be legally privileged. It is intended
> solely
> > for the addressee(s). Access to this internet electronic mail message by
> > anyone else is unauthorized. If you are not the intended recipient, any
> > disclosure, copying, distribution or any action taken or omitted to be
> > taken in reliance on it is prohibited and may be unlawful. The sender
> > believes that this E-mail and any attachments were free of any virus,
> worm,
> > Trojan horse, and/or malicious code when sent. This message and its
> > attachments could have been infected during transmission. By reading the
> > message and opening any attachments, the recipient accepts full
> > responsibility for taking protective and remedial action about viruses
> and
> > other defects. The sender's employer is not liable for any loss or damage
> > arising in any way.
> >
>
>
>
> --
> ~Vinit
>

Re: mesos/kafka issues (reviving old thread)

[Vinit Mahedia]

Justin,

When you say "working" - does it mean kafka-scheduler still has to be on
the same box as mesos-master? or you
have it working without that constraint?

On Wed, Jun 8, 2016 at 6:07 PM, Justin Ryan  wrote:

> inline
>
> On 6/8/16, 4:06 PM, "Justin Ryan"  wrote:
>
> >FYI, when I updated to the latest kafka-mesos (0.5.1.0) this problem went
> away.  FWIW, I’m actually using a branch which updates kafka to 0.10.0.0 as
> well:
> >
>
> Correction: 0.9.5.1 (current git master)
>
> > PR for kafka 0.10.0.0 (tests still fail, someone else did the bulk of
> porting but didn’t PR it) : https://github.com/mesos/kafka/pull/220
> >   ( ./gradlew jar –x test gets a successful build )
> >
> > Issue for the problem discussed in this thread:
> https://github.com/mesos/kafka/issues/199
> >
> >Cheers!
> >
>
>
> 
>
> P Please consider the environment before printing this e-mail
>
> The information in this electronic mail message is the sender's
> confidential business and may be legally privileged. It is intended solely
> for the addressee(s). Access to this internet electronic mail message by
> anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it is prohibited and may be unlawful. The sender
> believes that this E-mail and any attachments were free of any virus, worm,
> Trojan horse, and/or malicious code when sent. This message and its
> attachments could have been infected during transmission. By reading the
> message and opening any attachments, the recipient accepts full
> responsibility for taking protective and remedial action about viruses and
> other defects. The sender's employer is not liable for any loss or damage
> arising in any way.
>



-- 
~Vinit

Re: mesos/kafka issues (reviving old thread)

[Justin Ryan]

inline

On 6/8/16, 4:06 PM, "Justin Ryan"  wrote:

>FYI, when I updated to the latest kafka-mesos (0.5.1.0) this problem went 
>away.  FWIW, I’m actually using a branch which updates kafka to 0.10.0.0 as 
>well:
>

Correction: 0.9.5.1 (current git master)

> PR for kafka 0.10.0.0 (tests still fail, someone else did the bulk of porting 
> but didn’t PR it) : https://github.com/mesos/kafka/pull/220
>   ( ./gradlew jar –x test gets a successful build )
>
> Issue for the problem discussed in this thread: 
> https://github.com/mesos/kafka/issues/199
>
>Cheers!
>




P Please consider the environment before printing this e-mail

The information in this electronic mail message is the sender's confidential 
business and may be legally privileged. It is intended solely for the 
addressee(s). Access to this internet electronic mail message by anyone else is 
unauthorized. If you are not the intended recipient, any disclosure, copying, 
distribution or any action taken or omitted to be taken in reliance on it is 
prohibited and may be unlawful. The sender believes that this E-mail and any 
attachments were free of any virus, worm, Trojan horse, and/or malicious code 
when sent. This message and its attachments could have been infected during 
transmission. By reading the message and opening any attachments, the recipient 
accepts full responsibility for taking protective and remedial action about 
viruses and other defects. The sender's employer is not liable for any loss or 
damage arising in any way.

Re: mesos/kafka issues (reviving old thread)

[Justin Ryan]

FYI, when I updated to the latest kafka-mesos (0.5.1.0) this problem went away. 
 FWIW, I’m actually using a branch which updates kafka to 0.10.0.0 as well:

 PR for kafka 0.10.0.0 (tests still fail, someone else did the bulk of porting 
but didn’t PR it) : https://github.com/mesos/kafka/pull/220
   ( ./gradlew jar –x test gets a successful build )

 Issue for the problem discussed in this thread: 
https://github.com/mesos/kafka/issues/199

Cheers!

On 6/6/16, 12:23 PM, "Vinod Kone"  wrote:

>ZK communication is little different from framework communication. In the 
>former case, the ZK client library inside master opens up a connection to the 
>ZK server ensemble. In the latter case, the scheduler driver inside the 
>framework scheduler
> opens up a connection to the mesos master. But it is strange that marathon 
> from a different remote host works fine but kafka-mesos from a different 
> remote host has issues.
>
>On Mon, Jun 6, 2016 at 3:12 PM, Justin Ryan 
> wrote:
>
>inline
>
>On 6/6/16, 8:40 AM, "Vinod Kone"  wrote:
>
>>Have you tried running a different framework than the kafka one (maybe 
>>marathon? or chronos?) to rule out framework related issues? I'm surprised 
>>that it works when the scheduler and master are on the same host but not when 
>>they are different.
>> Looks like the request packets are getting dropped somewhere between the 
>> master NIC and the application.
>>
>
>Marathon is working just fine, I have flume running via Marathon and kafka 
>running via kafka-mesos.
>
>Yah, it does seem like the request packets are getting dropped, but that makes 
>no sense – how can the mesos-masters and zookeepers communicate with each 
>other, but an arbitrary other process can’t communicate with the active 
>mesos-master from another host which
> is a mesos-master? I’ve done a lot of packet inspection and can do more, but 
> overwhelmingly this seems to anchor the notion that there is no firewall in 
> play.
>
>
>
>P Please consider the environment before printing this e-mail
>
>The information in this electronic mail message is the sender's confidential 
>business and may be legally privileged. It is intended solely for the 
>addressee(s). Access to this internet electronic mail message by anyone else 
>is unauthorized. If you are not the
> intended recipient, any disclosure, copying, distribution or any action taken 
> or omitted to be taken in reliance on it is prohibited and may be unlawful. 
> The sender believes that this E-mail and any attachments were free of any 
> virus, worm, Trojan horse, and/or
> malicious code when sent. This message and its attachments could have been 
> infected during transmission. By reading the message and opening any 
> attachments, the recipient accepts full responsibility for taking protective 
> and remedial action about viruses and
> other defects. The sender's employer is not liable for any loss or damage 
> arising in any way.
>
>
>
>
>
>
>

Re: mesos/kafka issues (reviving old thread)

[Vinit Mahedia]

Vinod,

In my case, have two vagrant VMs, one where mesos-master runs and
frameworks on another.
This is my dev/debug setup where both marathon and kafka frameworks are
failing not just kafka.

But we do have a real (bare metal machines cluster) running marathon
successfully but kafka
framework fails there. I have tried both framework as a docker image and
running it directly on
machine.

I have the debugging environment setup, I see requests successfully
completed on master, it even
(thinks) it sends successfully, but framework never receives it. Nothing
can be seen on packet capture
as well which confuses me.


On Mon, Jun 6, 2016 at 2:23 PM, Vinod Kone  wrote:

> ZK communication is little different from framework communication. In the
> former case, the ZK client library inside master opens up a connection to
> the ZK server ensemble. In the latter case, the scheduler driver inside the
> framework scheduler opens up a connection to the mesos master. But it is
> strange that marathon from a different remote host works fine but
> kafka-mesos from a different remote host has issues.
>
> On Mon, Jun 6, 2016 at 3:12 PM, Justin Ryan  wrote:
>
> > inline
> >
> > On 6/6/16, 8:40 AM, "Vinod Kone"  wrote:
> >
> > >Have you tried running a different framework than the kafka one (maybe
> > marathon? or chronos?) to rule out framework related issues? I'm
> surprised
> > that it works when the scheduler and master are on the same host but not
> > when they are different.
> > > Looks like the request packets are getting dropped somewhere between
> the
> > master NIC and the application.
> > >
> >
> > Marathon is working just fine, I have flume running via Marathon and
> kafka
> > running via kafka-mesos.
> >
> > Yah, it does seem like the request packets are getting dropped, but that
> > makes no sense – how can the mesos-masters and zookeepers communicate
> with
> > each other, but an arbitrary other process can’t communicate with the
> > active mesos-master from another host which is a mesos-master? I’ve done
> a
> > lot of packet inspection and can do more, but overwhelmingly this seems
> to
> > anchor the notion that there is no firewall in play.
> >
> > 
> >
> > P Please consider the environment before printing this e-mail
> >
> > The information in this electronic mail message is the sender's
> > confidential business and may be legally privileged. It is intended
> solely
> > for the addressee(s). Access to this internet electronic mail message by
> > anyone else is unauthorized. If you are not the intended recipient, any
> > disclosure, copying, distribution or any action taken or omitted to be
> > taken in reliance on it is prohibited and may be unlawful. The sender
> > believes that this E-mail and any attachments were free of any virus,
> worm,
> > Trojan horse, and/or malicious code when sent. This message and its
> > attachments could have been infected during transmission. By reading the
> > message and opening any attachments, the recipient accepts full
> > responsibility for taking protective and remedial action about viruses
> and
> > other defects. The sender's employer is not liable for any loss or damage
> > arising in any way.
> >
>



-- 
~Vinit

Re: mesos/kafka issues (reviving old thread)

[Justin Ryan]

Right, mentioning ZK may have been a distraction.

My point was that the inactive mesos-master on the host running kafka-mesos 
scheduler is communicating with the active mesos-master.

To elaborate, this problem currently arises when a mesos-master failover 
occurs.  If I shut off the active master briefly and cause a failover back to 
the host running kafka-mesos, it starts working again.

The idea of sharing this was simply to elaborate that there’s no way port 5050 
of the mesos-master is inaccessible from the host running kafka-mesos, as mesos 
itself is communicating between those two hosts with that port.

On 6/6/16, 12:23 PM, "Vinod Kone"  wrote:

>ZK communication is little different from framework communication. In the 
>former case, the ZK client library inside master opens up a connection to the 
>ZK server ensemble. In the latter case, the scheduler driver inside the 
>framework scheduler
> opens up a connection to the mesos master. But it is strange that marathon 
> from a different remote host works fine but kafka-mesos from a different 
> remote host has issues.
>




P Please consider the environment before printing this e-mail

The information in this electronic mail message is the sender's confidential 
business and may be legally privileged. It is intended solely for the 
addressee(s). Access to this internet electronic mail message by anyone else is 
unauthorized. If you are not the intended recipient, any disclosure, copying, 
distribution or any action taken or omitted to be taken in reliance on it is 
prohibited and may be unlawful. The sender believes that this E-mail and any 
attachments were free of any virus, worm, Trojan horse, and/or malicious code 
when sent. This message and its attachments could have been infected during 
transmission. By reading the message and opening any attachments, the recipient 
accepts full responsibility for taking protective and remedial action about 
viruses and other defects. The sender's employer is not liable for any loss or 
damage arising in any way.

Re: mesos/kafka issues (reviving old thread)

[Vinod Kone]

ZK communication is little different from framework communication. In the
former case, the ZK client library inside master opens up a connection to
the ZK server ensemble. In the latter case, the scheduler driver inside the
framework scheduler opens up a connection to the mesos master. But it is
strange that marathon from a different remote host works fine but
kafka-mesos from a different remote host has issues.

On Mon, Jun 6, 2016 at 3:12 PM, Justin Ryan  wrote:

> inline
>
> On 6/6/16, 8:40 AM, "Vinod Kone"  wrote:
>
> >Have you tried running a different framework than the kafka one (maybe
> marathon? or chronos?) to rule out framework related issues? I'm surprised
> that it works when the scheduler and master are on the same host but not
> when they are different.
> > Looks like the request packets are getting dropped somewhere between the
> master NIC and the application.
> >
>
> Marathon is working just fine, I have flume running via Marathon and kafka
> running via kafka-mesos.
>
> Yah, it does seem like the request packets are getting dropped, but that
> makes no sense – how can the mesos-masters and zookeepers communicate with
> each other, but an arbitrary other process can’t communicate with the
> active mesos-master from another host which is a mesos-master? I’ve done a
> lot of packet inspection and can do more, but overwhelmingly this seems to
> anchor the notion that there is no firewall in play.
>
> 
>
> P Please consider the environment before printing this e-mail
>
> The information in this electronic mail message is the sender's
> confidential business and may be legally privileged. It is intended solely
> for the addressee(s). Access to this internet electronic mail message by
> anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it is prohibited and may be unlawful. The sender
> believes that this E-mail and any attachments were free of any virus, worm,
> Trojan horse, and/or malicious code when sent. This message and its
> attachments could have been infected during transmission. By reading the
> message and opening any attachments, the recipient accepts full
> responsibility for taking protective and remedial action about viruses and
> other defects. The sender's employer is not liable for any loss or damage
> arising in any way.
>

Re: mesos/kafka issues (reviving old thread)

[Justin Ryan]

inline

On 6/6/16, 8:40 AM, "Vinod Kone"  wrote:

>Have you tried running a different framework than the kafka one (maybe 
>marathon? or chronos?) to rule out framework related issues? I'm surprised 
>that it works when the scheduler and master are on the same host but not when 
>they are different.
> Looks like the request packets are getting dropped somewhere between the 
> master NIC and the application.
>

Marathon is working just fine, I have flume running via Marathon and kafka 
running via kafka-mesos.

Yah, it does seem like the request packets are getting dropped, but that makes 
no sense – how can the mesos-masters and zookeepers communicate with each 
other, but an arbitrary other process can’t communicate with the active 
mesos-master from another host which is a mesos-master? I’ve done a lot of 
packet inspection and can do more, but overwhelmingly this seems to anchor the 
notion that there is no firewall in play.



P Please consider the environment before printing this e-mail

The information in this electronic mail message is the sender's confidential 
business and may be legally privileged. It is intended solely for the 
addressee(s). Access to this internet electronic mail message by anyone else is 
unauthorized. If you are not the intended recipient, any disclosure, copying, 
distribution or any action taken or omitted to be taken in reliance on it is 
prohibited and may be unlawful. The sender believes that this E-mail and any 
attachments were free of any virus, worm, Trojan horse, and/or malicious code 
when sent. This message and its attachments could have been infected during 
transmission. By reading the message and opening any attachments, the recipient 
accepts full responsibility for taking protective and remedial action about 
viruses and other defects. The sender's employer is not liable for any loss or 
damage arising in any way.

Re: mesos/kafka issues (reviving old thread)

[Justin Ryan]

Hiya Vinit,

I’ve made some progress, I have a conditionally working setup, and another 
setup which was working now failing in new ways.

It does sound like your captures are similar to mine, what I found is that if I 
run the scheduler on, say, zk01 (which is also a mesos-master), while it is the 
leader, things work fine.  If mesos-master fails over to zk02 or zk03, the 
scheduler running on zk01 stops working, though its’ config points at all three 
machines.  Obviously this makes it difficult to run the scheduler itself as a 
mesos task.

Though I wish neither of us were having this problem, it’s good to not feel 
like I’m crazy, I kinda ran out of things to test until one day it occurred to 
me to check if it matters whether the scheduler runs on the active 
mesos-master.  I can think of a couple reasons this would be so, but haven’t 
had time to narrow it down.

Cheers!

Justin

On 6/3/16, 12:52 PM, "Vinit Mahedia"  wrote:

>
>
>​​Hey Justin,
>
>
>I am running in the same issues as you mentioned in this old
> thread 
> ,
>  did you resolve it?
>
>
>I see that kafka framework sends a POST request to register itself but mesos 
>master logs does not even show that the request was received, packet capture 
>does. My guess was that something is
> wrong in HTTP request but doing this same thing on local setup works fine so 
> that can't be the case. I can capture the requests on the master node so 
> there's no network issues either, just like in your case. I also verified 
> that two machines can communicate
> on using netcat as well.
>
>
>
>Like you mentioned in your thread, once it registered with mesos but that was 
>the only time that happened, the brokers did not start even then - "start" 
>timed out.
>
>
>^that behavior hints at possible bug in mesos, where it sits on this request 
>for too long and some times under some conditions, it gets through, that's 
>when it works.
>
>
>PS I have network capture showing POST from framework to master, if anyone 
>wants to take a look at it.
>
>
>--
>Vinit
>
>



P Please consider the environment before printing this e-mail

The information in this electronic mail message is the sender's confidential 
business and may be legally privileged. It is intended solely for the 
addressee(s). Access to this internet electronic mail message by anyone else is 
unauthorized. If you are not the intended recipient, any disclosure, copying, 
distribution or any action taken or omitted to be taken in reliance on it is 
prohibited and may be unlawful. The sender believes that this E-mail and any 
attachments were free of any virus, worm, Trojan horse, and/or malicious code 
when sent. This message and its attachments could have been infected during 
transmission. By reading the message and opening any attachments, the recipient 
accepts full responsibility for taking protective and remedial action about 
viruses and other defects. The sender's employer is not liable for any loss or 
damage arising in any way.

Re: mesos/kafka issues (reviving old thread)

[Justin Ryan]

inline

On 6/3/16, 2:19 PM, "Vinit Mahedia"  wrote:

>Justin,
>Yeah - as long as everything is on a single box (mesos-kafka scheduler, 
>mesos-master, zk etc.)
>things work just fine, which is what I meant by local setup.
>
>
>I did a local cluster setup as well, 3 vagrant machines, where it does
>not work. So it does not work at all
>if you have multi node setup (vagrant machines or bare metal).
>

I do have two working clusters, but the mesos-kafka scheduler must run on the 
active mesos-master leader, which may also be an indication of another symptom 
that just *happens* to coincide with this.

>
>others,
>
>Is there any alternative for getting kafka up and running on mesos  other than 
>this -
>mesos/kafka  ?
>

I thought about this for a while, but digging into the code, this is just an 
extension of MesosFramework, which is core mesos functionality.

We’re either missing something or have exposed a bug.

Of course, a person could just run kafka brokers with marathon, but I think 
this is worth the trouble and is fundamentally how things should be done.




P Please consider the environment before printing this e-mail

The information in this electronic mail message is the sender's confidential 
business and may be legally privileged. It is intended solely for the 
addressee(s). Access to this internet electronic mail message by anyone else is 
unauthorized. If you are not the intended recipient, any disclosure, copying, 
distribution or any action taken or omitted to be taken in reliance on it is 
prohibited and may be unlawful. The sender believes that this E-mail and any 
attachments were free of any virus, worm, Trojan horse, and/or malicious code 
when sent. This message and its attachments could have been infected during 
transmission. By reading the message and opening any attachments, the recipient 
accepts full responsibility for taking protective and remedial action about 
viruses and other defects. The sender's employer is not liable for any loss or 
damage arising in any way.

Re: mesos/kafka issues (reviving old thread)

[Vinod Kone]

Have you tried running a different framework than the kafka one (maybe
marathon? or chronos?) to rule out framework related issues? I'm surprised
that it works when the scheduler and master are on the same host but not
when they are different. Looks like the request packets are getting dropped
somewhere between the master NIC and the application.

On Fri, Jun 3, 2016 at 6:59 PM, Vinit Mahedia 
wrote:

> Justin,
>
> There's certainly a bug somewhere, either in mesos kafka framework or mesos
> itself. If I can get this
> mesos master running under debugger, this would be over in maybe few hours
> or few days.
>
> Kafka is a stateful service so can't run on marathon directly, there has to
> be some framework/wrapper
> which manages brokers - Read here
> <
> https://mesosphere.com/blog/2015/07/16/making-apache-kafka-elastic-with-apache-mesos/
> >
> .
>
> On Fri, Jun 3, 2016 at 4:36 PM, Justin Ryan  wrote:
>
> > inline
> >
> > On 6/3/16, 2:19 PM, "Vinit Mahedia"  wrote:
> >
> > >Justin,
> > >Yeah - as long as everything is on a single box (mesos-kafka scheduler,
> > mesos-master, zk etc.)
> > >things work just fine, which is what I meant by local setup.
> > >
> > >
> > >I did a local cluster setup as well, 3 vagrant machines, where it does
> > >not work. So it does not work at all
> > >if you have multi node setup (vagrant machines or bare metal).
> > >
> >
> > I do have two working clusters, but the mesos-kafka scheduler must run on
> > the active mesos-master leader, which may also be an indication of
> another
> > symptom that just *happens* to coincide with this.
> >
> > >
> > >others,
> > >
> > >Is there any alternative for getting kafka up and running on mesos
> other
> > than this -
> > >mesos/kafka  ?
> > >
> >
> > I thought about this for a while, but digging into the code, this is just
> > an extension of MesosFramework, which is core mesos functionality.
> >
> > We’re either missing something or have exposed a bug.
> >
> > Of course, a person could just run kafka brokers with marathon, but I
> > think this is worth the trouble and is fundamentally how things should be
> > done.
> >
> >
> > 
> >
> > P Please consider the environment before printing this e-mail
> >
> > The information in this electronic mail message is the sender's
> > confidential business and may be legally privileged. It is intended
> solely
> > for the addressee(s). Access to this internet electronic mail message by
> > anyone else is unauthorized. If you are not the intended recipient, any
> > disclosure, copying, distribution or any action taken or omitted to be
> > taken in reliance on it is prohibited and may be unlawful. The sender
> > believes that this E-mail and any attachments were free of any virus,
> worm,
> > Trojan horse, and/or malicious code when sent. This message and its
> > attachments could have been infected during transmission. By reading the
> > message and opening any attachments, the recipient accepts full
> > responsibility for taking protective and remedial action about viruses
> and
> > other defects. The sender's employer is not liable for any loss or damage
> > arising in any way.
> >
>
>
>
> --
> ~Vinit
>

Re: mesos/kafka issues (reviving old thread)

[Vinit Mahedia]

Justin,

There's certainly a bug somewhere, either in mesos kafka framework or mesos
itself. If I can get this
mesos master running under debugger, this would be over in maybe few hours
or few days.

Kafka is a stateful service so can't run on marathon directly, there has to
be some framework/wrapper
which manages brokers - Read here

.

On Fri, Jun 3, 2016 at 4:36 PM, Justin Ryan  wrote:

> inline
>
> On 6/3/16, 2:19 PM, "Vinit Mahedia"  wrote:
>
> >Justin,
> >Yeah - as long as everything is on a single box (mesos-kafka scheduler,
> mesos-master, zk etc.)
> >things work just fine, which is what I meant by local setup.
> >
> >
> >I did a local cluster setup as well, 3 vagrant machines, where it does
> >not work. So it does not work at all
> >if you have multi node setup (vagrant machines or bare metal).
> >
>
> I do have two working clusters, but the mesos-kafka scheduler must run on
> the active mesos-master leader, which may also be an indication of another
> symptom that just *happens* to coincide with this.
>
> >
> >others,
> >
> >Is there any alternative for getting kafka up and running on mesos  other
> than this -
> >mesos/kafka  ?
> >
>
> I thought about this for a while, but digging into the code, this is just
> an extension of MesosFramework, which is core mesos functionality.
>
> We’re either missing something or have exposed a bug.
>
> Of course, a person could just run kafka brokers with marathon, but I
> think this is worth the trouble and is fundamentally how things should be
> done.
>
>
> 
>
> P Please consider the environment before printing this e-mail
>
> The information in this electronic mail message is the sender's
> confidential business and may be legally privileged. It is intended solely
> for the addressee(s). Access to this internet electronic mail message by
> anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it is prohibited and may be unlawful. The sender
> believes that this E-mail and any attachments were free of any virus, worm,
> Trojan horse, and/or malicious code when sent. This message and its
> attachments could have been infected during transmission. By reading the
> message and opening any attachments, the recipient accepts full
> responsibility for taking protective and remedial action about viruses and
> other defects. The sender's employer is not liable for any loss or damage
> arising in any way.
>



-- 
~Vinit

Re: mesos/kafka issues (reviving old thread)

[Vinit Mahedia]

Justin,
Yeah - as long as everything is on a single box (mesos-kafka scheduler,
mesos-master, zk etc.)
things work just fine, which is what I meant by local setup.

I did a local cluster setup as well, 3 vagrant machines, where it does *not*
work. So it does not work at all
if you have multi node setup (vagrant machines or bare metal).

others,

Is there any alternative for getting kafka up and running on mesos  other
than this - mesos/kafka  ?



On Fri, Jun 3, 2016 at 3:53 PM, Justin Ryan  wrote:

> Hiya Vinit,
>
> I’ve made some progress, I have a conditionally working setup, and another
> setup which was working now failing in new ways.
>
> It does sound like your captures are similar to mine, what I found is that
> if I run the scheduler on, say, zk01 (which is also a mesos-master), while
> it is the leader, things work fine.  If mesos-master fails over to zk02 or
> zk03, the scheduler running on zk01 stops working, though its’ config
> points at all three machines.  Obviously this makes it difficult to run the
> scheduler itself as a mesos task.
>
> Though I wish neither of us were having this problem, it’s good to not
> feel like I’m crazy, I kinda ran out of things to test until one day it
> occurred to me to check if it matters whether the scheduler runs on the
> active mesos-master.  I can think of a couple reasons this would be so, but
> haven’t had time to narrow it down.
>
> Cheers!
>
> Justin
>
> On 6/3/16, 12:52 PM, "Vinit Mahedia"  wrote:
>
> >
> >
> >​​Hey Justin,
> >
> >
> >I am running in the same issues as you mentioned in this old
> > thread <
> https://mail-archives.apache.org/mod_mbox/mesos-user/201604.mbox/%3c89398c43-d45c-4653-8c0a-5ac987395...@ziprealty.com%3E>,
> did you resolve it?
> >
> >
> >I see that kafka framework sends a POST request to register itself but
> mesos master logs does not even show that the request was received, packet
> capture does. My guess was that something is
> > wrong in HTTP request but doing this same thing on local setup works
> fine so that can't be the case. I can capture the requests on the master
> node so there's no network issues either, just like in your case. I also
> verified that two machines can communicate
> > on using netcat as well.
> >
> >
> >
> >Like you mentioned in your thread, once it registered with mesos but that
> was the only time that happened, the brokers did not start even then -
> "start" timed out.
> >
> >
> >^that behavior hints at possible bug in mesos, where it sits on this
> request for too long and some times under some conditions, it gets through,
> that's when it works.
> >
> >
> >PS I have network capture showing POST from framework to master, if
> anyone wants to take a look at it.
> >
> >
> >--
> >Vinit
> >
> >
>
> 
>
> P Please consider the environment before printing this e-mail
>
> The information in this electronic mail message is the sender's
> confidential business and may be legally privileged. It is intended solely
> for the addressee(s). Access to this internet electronic mail message by
> anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it is prohibited and may be unlawful. The sender
> believes that this E-mail and any attachments were free of any virus, worm,
> Trojan horse, and/or malicious code when sent. This message and its
> attachments could have been infected during transmission. By reading the
> message and opening any attachments, the recipient accepts full
> responsibility for taking protective and remedial action about viruses and
> other defects. The sender's employer is not liable for any loss or damage
> arising in any way.
>



-- 
~Vinit

mesos/kafka issues (reviving old thread)

[Vinit Mahedia]

​​Hey Justin,

I am running in the same issues as you mentioned in this old thread
,
did you resolve it?

I see that kafka framework sends a POST request to register itself but
mesos master logs does not even show that the request was received, packet
capture does. My guess was that something is wrong in HTTP request but
doing this same thing on local setup works fine so that can't be the case.
I can capture the requests on the master node so there's no network issues
either, just like in your case. I also verified that two machines can
communicate on using netcat as well.

Like you mentioned in your thread, once it registered with mesos but that
was the only time that happened, the brokers did not start even then -
"start" timed out.

^that behavior hints at possible bug in mesos, where it sits on this
request for too long and some times under some conditions, it gets through,
that's when it works.

PS I have network capture showing POST from framework to master, if anyone
wants to take a look at it.

-- 
Vinit