Re: mesos/kafka issues (org.apache.mesos.Scheduler)

2016-04-12 Thread Justin Ryan 
Thanks to those who’ve taken the time to be helpful here, I dug into this a bit 
more today and verified some things:

  * When I run the scheduler on the active mesos master and tcpdump traffic to 
port 5050, I see none.
  * It does identify its’ IP as the running master at startup.
  * I do see zookeeper traffic in tcpdump
  * There is no kafka-mesos node in zookeeper

My config, for reference:

—
debug=true
user=marathon
storage=zk:/mesos-kafka
master=zk01:5050,zk02:5050,zk03:5050
zk=zk01:2181,zk02:2181,zk03:2181/kafka
api=http://zk01:7000
--

From: Justin Ryan <jur...@ziprealty.com<mailto:jur...@ziprealty.com>>
Reply-To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Date: Monday, April 11, 2016 at 1:46 PM
To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Subject: Re: mesos/kafka issues (org.apache.mesos.Scheduler)

I have 3 hosts running zookeeper, mesos masters, and marathon, an HDFS 
namenode, and 10 worker nodes running mesos-slave and HDFS datanodes.

I don’t remember having set LIBPROCESS_IP in the past, maybe it’s part of some 
slightly newer code, so I went ahead and did this and verified 
MESOS_NATIVE_JAVA_LIBRARY as well, no change.

Shouldn’t be any firewall rules, and like I said, at least one of these 
clusters I built with chef a couple of months back.  I had some concern that 
when I launched the production cluster, I may have inadvertently copied some 
config related to the testing environment (e.g. wrong zk hosts), but I’ve 
re-verified this all a number of times, and see no indication that hosts from 
one are talking to the other.

From: Kevin Lu <ke...@netsil.com<mailto:ke...@netsil.com>>
Reply-To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Date: Monday, April 11, 2016 at 1:40 PM
To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Subject: Re: mesos/kafka issues (org.apache.mesos.Scheduler)

What's your master/slave setup? What are your ports/firewall rules? In the 
past, when weird situations like these happen to me, it's usually because of 
some firewall rule, and at that point I'll ssh into the machine where my 
framework is running and see what ports it's trying to talk to via netstat.

I'm sure you've done this as well, but the github docs do say to set the 
LIBPROCESS_IP environment variable, if you haven't done so.

On Mon, Apr 11, 2016 at 1:35 PM, Justin Ryan 
<jur...@ziprealty.com<mailto:jur...@ziprealty.com>> wrote:
I have tried it without the &&, and i can ‘broker stop’, then ‘broker start’ 
with no change, though I’ll make sure on my next zk clear to be sure to try 
without the &&

I am, indeed, not seeing the framework at all, and when this happens, the last 
line of output running the scheduler is:

  I0411 13:34:37.174973 14368 sched.cpp:336] No credentials provided. 
Attempting to register without authentication

When it works, the next step is basically, ‘registered framework 
--XXX-XXX'

From: Kevin Lu <ke...@netsil.com<mailto:ke...@netsil.com>>
Reply-To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Date: Monday, April 11, 2016 at 1:33 PM
To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Subject: Re: mesos/kafka issues (org.apache.mesos.Scheduler)

Can you try it step-by-step without the "&&"?

Also, IIRC, mesos creates separate tasks for the kafka framework and the 
broker. Are you not even seeing the framework in the mesos UI?

On Mon, Apr 11, 2016 at 1:29 PM, Justin Ryan 
<jur...@ziprealty.com<mailto:jur...@ziprealty.com>> wrote:
Hi, folks!

In pursuit of a mesos-based solution for distributed logging and processing, 
I’ve been experimenting with the mesos/kafka project 
(github.com/mesos/kafka<http://github.com/mesos/kafka>).  I had great success 
for some weeks during initial testing, and am now having trouble getting 
brokers to launch at all.

This code has been adopted by the mesos project, but also as far as I can tell 
the meat of it relies on functionality from org.apache.mesos by implementing 
the Scheduler interface.

Let’s say, for instance, I run:

  ./kafka-mesos.sh broker add 0..2 --options 
log.retention.hours=1,log.retention.bytes.per.topic=1073741824 && 
./kafka-mesos.sh broker start 0..2

The broker start simply times out and status never changes.  I’ve cleared ZK a 
number of times, which is the way I’ve been advised to get out of wierd mesos 
states in the past.  The mesos UI ne

Re: mesos/kafka issues (org.apache.mesos.Scheduler)

2016-04-11 Thread Justin Ryan 
I’m now using /var/mesos as my work_dir, and don’t have any logs from when 
they’ve gone missing because getting them to start hasn’t happened for so long. 
:/

From: Greg Mann <g...@mesosphere.io<mailto:g...@mesosphere.io>>
Reply-To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Date: Monday, April 11, 2016 at 1:46 PM
To: user <user@mesos.apache.org<mailto:user@mesos.apache.org>>
Subject: Re: mesos/kafka issues (org.apache.mesos.Scheduler)

Hi Justin,
Do you have master/agent logs from a time when these tasks would have gone 
missing from the Mesos UI?

What location are you using for the work_dir on the agents?

Cheers,
Greg


On Mon, Apr 11, 2016 at 1:41 PM, Justin Ryan 
<jur...@ziprealty.com<mailto:jur...@ziprealty.com>> wrote:
Update : I noticed one of the clusters had a framework registered before I 
cleared ZK, but is now seeing the same failure at scheduler start.

When the brokers do launch, in recent times, they disappear from mesos within a 
day, although they keep running.  I have another thread on this list talking 
about that which it’s unclear if is directly related – I had the same happen to 
flume launched by marathon.

From: Justin Ryan <jur...@ziprealty.com<mailto:jur...@ziprealty.com>>
Reply-To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Date: Monday, April 11, 2016 at 1:35 PM

To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Subject: Re: mesos/kafka issues (org.apache.mesos.Scheduler)

I have tried it without the &&, and i can ‘broker stop’, then ‘broker start’ 
with no change, though I’ll make sure on my next zk clear to be sure to try 
without the &&

I am, indeed, not seeing the framework at all, and when this happens, the last 
line of output running the scheduler is:

  I0411 13:34:37.174973 14368 sched.cpp:336] No credentials provided. 
Attempting to register without authentication

When it works, the next step is basically, ‘registered framework 
--XXX-XXX'

From: Kevin Lu <ke...@netsil.com<mailto:ke...@netsil.com>>
Reply-To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Date: Monday, April 11, 2016 at 1:33 PM
To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Subject: Re: mesos/kafka issues (org.apache.mesos.Scheduler)

Can you try it step-by-step without the "&&"?

Also, IIRC, mesos creates separate tasks for the kafka framework and the 
broker. Are you not even seeing the framework in the mesos UI?

On Mon, Apr 11, 2016 at 1:29 PM, Justin Ryan 
<jur...@ziprealty.com<mailto:jur...@ziprealty.com>> wrote:
Hi, folks!

In pursuit of a mesos-based solution for distributed logging and processing, 
I’ve been experimenting with the mesos/kafka project 
(github.com/mesos/kafka<http://github.com/mesos/kafka>).  I had great success 
for some weeks during initial testing, and am now having trouble getting 
brokers to launch at all.

This code has been adopted by the mesos project, but also as far as I can tell 
the meat of it relies on functionality from org.apache.mesos by implementing 
the Scheduler interface.

Let’s say, for instance, I run:

  ./kafka-mesos.sh broker add 0..2 --options 
log.retention.hours=1,log.retention.bytes.per.topic=1073741824 && 
./kafka-mesos.sh broker start 0..2

The broker start simply times out and status never changes.  I’ve cleared ZK a 
number of times, which is the way I’ve been advised to get out of wierd mesos 
states in the past.  The mesos UI never shows an job in STARTING or other 
state, or a failed / ended job.

Any idea what I might be running into? This was working consistently for weeks 
on end and recently stopped working altogether about 95% of the time.  When it 
works, it only sporadically works.

TIA,

JR


P Please consider the environment before printing this e-mail

The information in this electronic mail message is the sender's confidential 
business and may be legally privileged. It is intended solely for the 
addressee(s). Access to this internet electronic mail message by anyone else is 
unauthorized. If you are not the intended recipient, any disclosure, copying, 
distribution or any action taken or omitted to be taken in reliance on it is 
prohibited and may be unlawful. The sender believes that this E-mail and any 
attachments were free of any virus, worm, Trojan horse, and/or malicious code 
when sent. This message and its attachments could have been infected during 
transmission. By reading the message and opening any attachments, the recipient 
accepts full responsibility for taking protective and remedial action about 
viruses and other defects. The sender's employer is not liable for any loss or 
damage arising in any way.

Re: mesos/kafka issues (org.apache.mesos.Scheduler)

2016-04-11 Thread Greg Mann 
Hi Justin,
Do you have master/agent logs from a time when these tasks would have gone
missing from the Mesos UI?

What location are you using for the work_dir on the agents?

Cheers,
Greg


On Mon, Apr 11, 2016 at 1:41 PM, Justin Ryan <jur...@ziprealty.com> wrote:

> Update : I noticed one of the clusters had a framework registered before I
> cleared ZK, but is now seeing the same failure at scheduler start.
>
> When the brokers do launch, in recent times, they disappear from mesos
> within a day, although they keep running.  I have another thread on this
> list talking about that which it’s unclear if is directly related – I had
> the same happen to flume launched by marathon.
>
> From: Justin Ryan <jur...@ziprealty.com>
> Reply-To: "user@mesos.apache.org" <user@mesos.apache.org>
> Date: Monday, April 11, 2016 at 1:35 PM
>
> To: "user@mesos.apache.org" <user@mesos.apache.org>
> Subject: Re: mesos/kafka issues (org.apache.mesos.Scheduler)
>
> I have tried it without the &&, and i can ‘broker stop’, then ‘broker
> start’ with no change, though I’ll make sure on my next zk clear to be sure
> to try without the &&
>
> I am, indeed, not seeing the framework at all, and when this happens, the
> last line of output running the scheduler is:
>
>   I0411 13:34:37.174973 14368 sched.cpp:336] No credentials provided.
> Attempting to register without authentication
>
> When it works, the next step is basically, ‘registered framework
> --XXX-XXX'
>
> From: Kevin Lu <ke...@netsil.com>
> Reply-To: "user@mesos.apache.org" <user@mesos.apache.org>
> Date: Monday, April 11, 2016 at 1:33 PM
> To: "user@mesos.apache.org" <user@mesos.apache.org>
> Subject: Re: mesos/kafka issues (org.apache.mesos.Scheduler)
>
> Can you try it step-by-step without the "&&"?
>
> Also, IIRC, mesos creates separate tasks for the kafka framework and the
> broker. Are you not even seeing the framework in the mesos UI?
>
> On Mon, Apr 11, 2016 at 1:29 PM, Justin Ryan <jur...@ziprealty.com> wrote:
>
>> Hi, folks!
>>
>> In pursuit of a mesos-based solution for distributed logging and
>> processing, I’ve been experimenting with the mesos/kafka project (
>> github.com/mesos/kafka).  I had great success for some weeks during
>> initial testing, and am now having trouble getting brokers to launch at all.
>>
>> This code has been adopted by the mesos project, but also as far as I can
>> tell the meat of it relies on functionality from org.apache.mesos by
>> implementing the Scheduler interface.
>>
>> Let’s say, for instance, I run:
>>
>>   ./kafka-mesos.sh broker add 0..2 --options
>> log.retention.hours=1,log.retention.bytes.per.topic=1073741824 &&
>> ./kafka-mesos.sh broker start 0..2
>>
>> The broker start simply times out and status never changes.  I’ve cleared
>> ZK a number of times, which is the way I’ve been advised to get out of
>> wierd mesos states in the past.  The mesos UI never shows an job in
>> STARTING or other state, or a failed / ended job.
>>
>> Any idea what I might be running into? This was working consistently for
>> weeks on end and recently stopped working altogether about 95% of the
>> time.  When it works, it only sporadically works.
>>
>> TIA,
>>
>> JR
>> --
>>
>> P Please consider the environment before printing this e-mail
>> The information in this electronic mail message is the sender's
>> confidential business and may be legally privileged. It is intended solely
>> for the addressee(s). Access to this internet electronic mail message by
>> anyone else is unauthorized. If you are not the intended recipient, any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it is prohibited and may be unlawful. The sender
>> believes that this E-mail and any attachments were free of any virus, worm,
>> Trojan horse, and/or malicious code when sent. This message and its
>> attachments could have been infected during transmission. By reading the
>> message and opening any attachments, the recipient accepts full
>> responsibility for taking protective and remedial action about viruses and
>> other defects. The sender's employer is not liable for any loss or damage
>> arising in any way.
>>
>
>

Re: mesos/kafka issues (org.apache.mesos.Scheduler)

2016-04-11 Thread Justin Ryan 
I have 3 hosts running zookeeper, mesos masters, and marathon, an HDFS 
namenode, and 10 worker nodes running mesos-slave and HDFS datanodes.

I don’t remember having set LIBPROCESS_IP in the past, maybe it’s part of some 
slightly newer code, so I went ahead and did this and verified 
MESOS_NATIVE_JAVA_LIBRARY as well, no change.

Shouldn’t be any firewall rules, and like I said, at least one of these 
clusters I built with chef a couple of months back.  I had some concern that 
when I launched the production cluster, I may have inadvertently copied some 
config related to the testing environment (e.g. wrong zk hosts), but I’ve 
re-verified this all a number of times, and see no indication that hosts from 
one are talking to the other.

From: Kevin Lu <ke...@netsil.com<mailto:ke...@netsil.com>>
Reply-To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Date: Monday, April 11, 2016 at 1:40 PM
To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Subject: Re: mesos/kafka issues (org.apache.mesos.Scheduler)

What's your master/slave setup? What are your ports/firewall rules? In the 
past, when weird situations like these happen to me, it's usually because of 
some firewall rule, and at that point I'll ssh into the machine where my 
framework is running and see what ports it's trying to talk to via netstat.

I'm sure you've done this as well, but the github docs do say to set the 
LIBPROCESS_IP environment variable, if you haven't done so.

On Mon, Apr 11, 2016 at 1:35 PM, Justin Ryan 
<jur...@ziprealty.com<mailto:jur...@ziprealty.com>> wrote:
I have tried it without the &&, and i can ‘broker stop’, then ‘broker start’ 
with no change, though I’ll make sure on my next zk clear to be sure to try 
without the &&

I am, indeed, not seeing the framework at all, and when this happens, the last 
line of output running the scheduler is:

  I0411 13:34:37.174973 14368 sched.cpp:336] No credentials provided. 
Attempting to register without authentication

When it works, the next step is basically, ‘registered framework 
--XXX-XXX'

From: Kevin Lu <ke...@netsil.com<mailto:ke...@netsil.com>>
Reply-To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Date: Monday, April 11, 2016 at 1:33 PM
To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Subject: Re: mesos/kafka issues (org.apache.mesos.Scheduler)

Can you try it step-by-step without the "&&"?

Also, IIRC, mesos creates separate tasks for the kafka framework and the 
broker. Are you not even seeing the framework in the mesos UI?

On Mon, Apr 11, 2016 at 1:29 PM, Justin Ryan 
<jur...@ziprealty.com<mailto:jur...@ziprealty.com>> wrote:
Hi, folks!

In pursuit of a mesos-based solution for distributed logging and processing, 
I’ve been experimenting with the mesos/kafka project 
(github.com/mesos/kafka<http://github.com/mesos/kafka>).  I had great success 
for some weeks during initial testing, and am now having trouble getting 
brokers to launch at all.

This code has been adopted by the mesos project, but also as far as I can tell 
the meat of it relies on functionality from org.apache.mesos by implementing 
the Scheduler interface.

Let’s say, for instance, I run:

  ./kafka-mesos.sh broker add 0..2 --options 
log.retention.hours=1,log.retention.bytes.per.topic=1073741824 && 
./kafka-mesos.sh broker start 0..2

The broker start simply times out and status never changes.  I’ve cleared ZK a 
number of times, which is the way I’ve been advised to get out of wierd mesos 
states in the past.  The mesos UI never shows an job in STARTING or other 
state, or a failed / ended job.

Any idea what I might be running into? This was working consistently for weeks 
on end and recently stopped working altogether about 95% of the time.  When it 
works, it only sporadically works.

TIA,

JR


P Please consider the environment before printing this e-mail

The information in this electronic mail message is the sender's confidential 
business and may be legally privileged. It is intended solely for the 
addressee(s). Access to this internet electronic mail message by anyone else is 
unauthorized. If you are not the intended recipient, any disclosure, copying, 
distribution or any action taken or omitted to be taken in reliance on it is 
prohibited and may be unlawful. The sender believes that this E-mail and any 
attachments were free of any virus, worm, Trojan horse, and/or malicious code 
when sent. This message and its attachments could have been infected during 
transmission. By reading the message and ope

Re: mesos/kafka issues (org.apache.mesos.Scheduler)

2016-04-11 Thread Justin Ryan 
Update : I noticed one of the clusters had a framework registered before I 
cleared ZK, but is now seeing the same failure at scheduler start.

When the brokers do launch, in recent times, they disappear from mesos within a 
day, although they keep running.  I have another thread on this list talking 
about that which it’s unclear if is directly related – I had the same happen to 
flume launched by marathon.

From: Justin Ryan <jur...@ziprealty.com<mailto:jur...@ziprealty.com>>
Reply-To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Date: Monday, April 11, 2016 at 1:35 PM
To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Subject: Re: mesos/kafka issues (org.apache.mesos.Scheduler)

I have tried it without the &&, and i can ‘broker stop’, then ‘broker start’ 
with no change, though I’ll make sure on my next zk clear to be sure to try 
without the &&

I am, indeed, not seeing the framework at all, and when this happens, the last 
line of output running the scheduler is:

  I0411 13:34:37.174973 14368 sched.cpp:336] No credentials provided. 
Attempting to register without authentication

When it works, the next step is basically, ‘registered framework 
--XXX-XXX'

From: Kevin Lu <ke...@netsil.com<mailto:ke...@netsil.com>>
Reply-To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Date: Monday, April 11, 2016 at 1:33 PM
To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Subject: Re: mesos/kafka issues (org.apache.mesos.Scheduler)

Can you try it step-by-step without the "&&"?

Also, IIRC, mesos creates separate tasks for the kafka framework and the 
broker. Are you not even seeing the framework in the mesos UI?

On Mon, Apr 11, 2016 at 1:29 PM, Justin Ryan 
<jur...@ziprealty.com<mailto:jur...@ziprealty.com>> wrote:
Hi, folks!

In pursuit of a mesos-based solution for distributed logging and processing, 
I’ve been experimenting with the mesos/kafka project 
(github.com/mesos/kafka<http://github.com/mesos/kafka>).  I had great success 
for some weeks during initial testing, and am now having trouble getting 
brokers to launch at all.

This code has been adopted by the mesos project, but also as far as I can tell 
the meat of it relies on functionality from org.apache.mesos by implementing 
the Scheduler interface.

Let’s say, for instance, I run:

  ./kafka-mesos.sh broker add 0..2 --options 
log.retention.hours=1,log.retention.bytes.per.topic=1073741824 && 
./kafka-mesos.sh broker start 0..2

The broker start simply times out and status never changes.  I’ve cleared ZK a 
number of times, which is the way I’ve been advised to get out of wierd mesos 
states in the past.  The mesos UI never shows an job in STARTING or other 
state, or a failed / ended job.

Any idea what I might be running into? This was working consistently for weeks 
on end and recently stopped working altogether about 95% of the time.  When it 
works, it only sporadically works.

TIA,

JR


P Please consider the environment before printing this e-mail

The information in this electronic mail message is the sender's confidential 
business and may be legally privileged. It is intended solely for the 
addressee(s). Access to this internet electronic mail message by anyone else is 
unauthorized. If you are not the intended recipient, any disclosure, copying, 
distribution or any action taken or omitted to be taken in reliance on it is 
prohibited and may be unlawful. The sender believes that this E-mail and any 
attachments were free of any virus, worm, Trojan horse, and/or malicious code 
when sent. This message and its attachments could have been infected during 
transmission. By reading the message and opening any attachments, the recipient 
accepts full responsibility for taking protective and remedial action about 
viruses and other defects. The sender's employer is not liable for any loss or 
damage arising in any way.

Re: mesos/kafka issues (org.apache.mesos.Scheduler)

2016-04-11 Thread Kevin Lu 
What's your master/slave setup? What are your ports/firewall rules? In the
past, when weird situations like these happen to me, it's usually because
of some firewall rule, and at that point I'll ssh into the machine where my
framework is running and see what ports it's trying to talk to via netstat.

I'm sure you've done this as well, but the github docs do say to set the
LIBPROCESS_IP environment variable, if you haven't done so.

On Mon, Apr 11, 2016 at 1:35 PM, Justin Ryan <jur...@ziprealty.com> wrote:

> I have tried it without the &&, and i can ‘broker stop’, then ‘broker
> start’ with no change, though I’ll make sure on my next zk clear to be sure
> to try without the &&
>
> I am, indeed, not seeing the framework at all, and when this happens, the
> last line of output running the scheduler is:
>
>   I0411 13:34:37.174973 14368 sched.cpp:336] No credentials provided.
> Attempting to register without authentication
>
> When it works, the next step is basically, ‘registered framework
> --XXX-XXX'
>
> From: Kevin Lu <ke...@netsil.com>
> Reply-To: "user@mesos.apache.org" <user@mesos.apache.org>
> Date: Monday, April 11, 2016 at 1:33 PM
> To: "user@mesos.apache.org" <user@mesos.apache.org>
> Subject: Re: mesos/kafka issues (org.apache.mesos.Scheduler)
>
> Can you try it step-by-step without the "&&"?
>
> Also, IIRC, mesos creates separate tasks for the kafka framework and the
> broker. Are you not even seeing the framework in the mesos UI?
>
> On Mon, Apr 11, 2016 at 1:29 PM, Justin Ryan <jur...@ziprealty.com> wrote:
>
>> Hi, folks!
>>
>> In pursuit of a mesos-based solution for distributed logging and
>> processing, I’ve been experimenting with the mesos/kafka project (
>> github.com/mesos/kafka).  I had great success for some weeks during
>> initial testing, and am now having trouble getting brokers to launch at all.
>>
>> This code has been adopted by the mesos project, but also as far as I can
>> tell the meat of it relies on functionality from org.apache.mesos by
>> implementing the Scheduler interface.
>>
>> Let’s say, for instance, I run:
>>
>>   ./kafka-mesos.sh broker add 0..2 --options
>> log.retention.hours=1,log.retention.bytes.per.topic=1073741824 &&
>> ./kafka-mesos.sh broker start 0..2
>>
>> The broker start simply times out and status never changes.  I’ve cleared
>> ZK a number of times, which is the way I’ve been advised to get out of
>> wierd mesos states in the past.  The mesos UI never shows an job in
>> STARTING or other state, or a failed / ended job.
>>
>> Any idea what I might be running into? This was working consistently for
>> weeks on end and recently stopped working altogether about 95% of the
>> time.  When it works, it only sporadically works.
>>
>> TIA,
>>
>> JR
>> --
>>
>> P Please consider the environment before printing this e-mail
>> The information in this electronic mail message is the sender's
>> confidential business and may be legally privileged. It is intended solely
>> for the addressee(s). Access to this internet electronic mail message by
>> anyone else is unauthorized. If you are not the intended recipient, any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it is prohibited and may be unlawful. The sender
>> believes that this E-mail and any attachments were free of any virus, worm,
>> Trojan horse, and/or malicious code when sent. This message and its
>> attachments could have been infected during transmission. By reading the
>> message and opening any attachments, the recipient accepts full
>> responsibility for taking protective and remedial action about viruses and
>> other defects. The sender's employer is not liable for any loss or damage
>> arising in any way.
>>
>
>

Re: mesos/kafka issues (org.apache.mesos.Scheduler)

2016-04-11 Thread Justin Ryan 
I have tried it without the &&, and i can ‘broker stop’, then ‘broker start’ 
with no change, though I’ll make sure on my next zk clear to be sure to try 
without the &&

I am, indeed, not seeing the framework at all, and when this happens, the last 
line of output running the scheduler is:

  I0411 13:34:37.174973 14368 sched.cpp:336] No credentials provided. 
Attempting to register without authentication

When it works, the next step is basically, ‘registered framework 
--XXX-XXX'

From: Kevin Lu <ke...@netsil.com<mailto:ke...@netsil.com>>
Reply-To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Date: Monday, April 11, 2016 at 1:33 PM
To: "user@mesos.apache.org<mailto:user@mesos.apache.org>" 
<user@mesos.apache.org<mailto:user@mesos.apache.org>>
Subject: Re: mesos/kafka issues (org.apache.mesos.Scheduler)

Can you try it step-by-step without the "&&"?

Also, IIRC, mesos creates separate tasks for the kafka framework and the 
broker. Are you not even seeing the framework in the mesos UI?

On Mon, Apr 11, 2016 at 1:29 PM, Justin Ryan 
<jur...@ziprealty.com<mailto:jur...@ziprealty.com>> wrote:
Hi, folks!

In pursuit of a mesos-based solution for distributed logging and processing, 
I’ve been experimenting with the mesos/kafka project 
(github.com/mesos/kafka<http://github.com/mesos/kafka>).  I had great success 
for some weeks during initial testing, and am now having trouble getting 
brokers to launch at all.

This code has been adopted by the mesos project, but also as far as I can tell 
the meat of it relies on functionality from org.apache.mesos by implementing 
the Scheduler interface.

Let’s say, for instance, I run:

  ./kafka-mesos.sh broker add 0..2 --options 
log.retention.hours=1,log.retention.bytes.per.topic=1073741824 && 
./kafka-mesos.sh broker start 0..2

The broker start simply times out and status never changes.  I’ve cleared ZK a 
number of times, which is the way I’ve been advised to get out of wierd mesos 
states in the past.  The mesos UI never shows an job in STARTING or other 
state, or a failed / ended job.

Any idea what I might be running into? This was working consistently for weeks 
on end and recently stopped working altogether about 95% of the time.  When it 
works, it only sporadically works.

TIA,

JR


P Please consider the environment before printing this e-mail

The information in this electronic mail message is the sender's confidential 
business and may be legally privileged. It is intended solely for the 
addressee(s). Access to this internet electronic mail message by anyone else is 
unauthorized. If you are not the intended recipient, any disclosure, copying, 
distribution or any action taken or omitted to be taken in reliance on it is 
prohibited and may be unlawful. The sender believes that this E-mail and any 
attachments were free of any virus, worm, Trojan horse, and/or malicious code 
when sent. This message and its attachments could have been infected during 
transmission. By reading the message and opening any attachments, the recipient 
accepts full responsibility for taking protective and remedial action about 
viruses and other defects. The sender's employer is not liable for any loss or 
damage arising in any way.

Re: mesos/kafka issues (org.apache.mesos.Scheduler)

2016-04-11 Thread Kevin Lu 
Can you try it step-by-step without the "&&"?

Also, IIRC, mesos creates separate tasks for the kafka framework and the
broker. Are you not even seeing the framework in the mesos UI?

On Mon, Apr 11, 2016 at 1:29 PM, Justin Ryan  wrote:

> Hi, folks!
>
> In pursuit of a mesos-based solution for distributed logging and
> processing, I’ve been experimenting with the mesos/kafka project (
> github.com/mesos/kafka).  I had great success for some weeks during
> initial testing, and am now having trouble getting brokers to launch at all.
>
> This code has been adopted by the mesos project, but also as far as I can
> tell the meat of it relies on functionality from org.apache.mesos by
> implementing the Scheduler interface.
>
> Let’s say, for instance, I run:
>
>   ./kafka-mesos.sh broker add 0..2 --options
> log.retention.hours=1,log.retention.bytes.per.topic=1073741824 &&
> ./kafka-mesos.sh broker start 0..2
>
> The broker start simply times out and status never changes.  I’ve cleared
> ZK a number of times, which is the way I’ve been advised to get out of
> wierd mesos states in the past.  The mesos UI never shows an job in
> STARTING or other state, or a failed / ended job.
>
> Any idea what I might be running into? This was working consistently for
> weeks on end and recently stopped working altogether about 95% of the
> time.  When it works, it only sporadically works.
>
> TIA,
>
> JR
> --
>
> P Please consider the environment before printing this e-mail
> The information in this electronic mail message is the sender's
> confidential business and may be legally privileged. It is intended solely
> for the addressee(s). Access to this internet electronic mail message by
> anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it is prohibited and may be unlawful. The sender
> believes that this E-mail and any attachments were free of any virus, worm,
> Trojan horse, and/or malicious code when sent. This message and its
> attachments could have been infected during transmission. By reading the
> message and opening any attachments, the recipient accepts full
> responsibility for taking protective and remedial action about viruses and
> other defects. The sender's employer is not liable for any loss or damage
> arising in any way.
>