Re: Weird Spark Dispatcher Offers?

[Tim Chen]

Do you have jobs enqueued? And if none of the jobs matches any offer it
will just decline it.

What's your job resource specifications?

Tim

On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite 
wrote:

> Hey All,
>
> Using spark with mesos and docker.
>
> I'm wondering if anybody's seen the behavior of spark dispatcher where it
> just continually requests resources and immediately declines the offer.
>
> https://gist.github.com/anonymous/41e7c91899b0122b91a7
>
> I'm trying to debug some issues with spark and I'm having trouble figuring
> out if this is part of the problem or if it's safe to ignore it.
>
> Any help or pointers would be appreciated.
>
> Thanks!
> - Alan
>

Re: Weird Spark Dispatcher Offers?

[Alan Braithwaite]

>
> So if there is no jobs to run the dispatcher will decline all offers by
> default.
>

So would this be a bug in mesos then?  I'm not sure I understand how this
offer is appearing in the first place.  It only shows up in the master logs
when I start the dispatcher.


> Also we list all the jobs enqueued and it's specifications in the Spark
> dispatcher UI, you should see the port in the dispatcher logs itself.


Yes, this job is not listed under that UI.  Hence my confusion.

Thanks,
- Alan

On Fri, Oct 2, 2015 at 11:49 AM, Tim Chen  wrote:

> So if there is no jobs to run the dispatcher will decline all offers by
> default.
>
> Also we list all the jobs enqueued and it's specifications in the Spark
> dispatcher UI, you should see the port in the dispatcher logs itself.
>
> Tim
>
> On Fri, Oct 2, 2015 at 11:46 AM, Alan Braithwaite 
> wrote:
>
>> This happened right after blowing away /var/lib/mesos zk://mesos and
>> zk://spark_mesos_dispatcher and before I've submitted anything new to it so
>> I _shouldn't_ have anything enqueued.  Unless there's state being stored
>> somewhere besides those places that I don't know about.
>>
>> I'm not sure what the resource specifications are for this one because I
>> didn't submit it directly.  If you have a way for me to grab a specific
>> offer configuration, I'd be delighted to provide it.  I just can't seem to
>> figure out how to get that information after digging through the mesos docs
>> :-(
>>
>> Also, I can't read the docker logs because:
>>
>> Oct 02 11:39:59 sparky docker[556]:
>> time="2015-10-02T11:39:59.165474049-07:00" level=error msg="Error streaming
>> logs: invalid character '\\x00' looking for beginning of value"
>>
>> (that's coming from the spark-dispatcher docker).
>>
>> Thanks!
>> - Alan
>>
>> On Fri, Oct 2, 2015 at 11:36 AM, Tim Chen  wrote:
>>
>>> Do you have jobs enqueued? And if none of the jobs matches any offer it
>>> will just decline it.
>>>
>>> What's your job resource specifications?
>>>
>>> Tim
>>>
>>> On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite 
>>> wrote:
>>>
 Hey All,

 Using spark with mesos and docker.

 I'm wondering if anybody's seen the behavior of spark dispatcher where
 it just continually requests resources and immediately declines the offer.

 https://gist.github.com/anonymous/41e7c91899b0122b91a7

 I'm trying to debug some issues with spark and I'm having trouble
 figuring out if this is part of the problem or if it's safe to ignore it.

 Any help or pointers would be appreciated.

 Thanks!
 - Alan

>>>
>>>
>>
>

Re: Weird Spark Dispatcher Offers?

[Alan Braithwaite]

This happened right after blowing away /var/lib/mesos zk://mesos and
zk://spark_mesos_dispatcher and before I've submitted anything new to it so
I _shouldn't_ have anything enqueued.  Unless there's state being stored
somewhere besides those places that I don't know about.

I'm not sure what the resource specifications are for this one because I
didn't submit it directly.  If you have a way for me to grab a specific
offer configuration, I'd be delighted to provide it.  I just can't seem to
figure out how to get that information after digging through the mesos docs
:-(

Also, I can't read the docker logs because:

Oct 02 11:39:59 sparky docker[556]:
time="2015-10-02T11:39:59.165474049-07:00" level=error msg="Error streaming
logs: invalid character '\\x00' looking for beginning of value"

(that's coming from the spark-dispatcher docker).

Thanks!
- Alan

On Fri, Oct 2, 2015 at 11:36 AM, Tim Chen  wrote:

> Do you have jobs enqueued? And if none of the jobs matches any offer it
> will just decline it.
>
> What's your job resource specifications?
>
> Tim
>
> On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite 
> wrote:
>
>> Hey All,
>>
>> Using spark with mesos and docker.
>>
>> I'm wondering if anybody's seen the behavior of spark dispatcher where it
>> just continually requests resources and immediately declines the offer.
>>
>> https://gist.github.com/anonymous/41e7c91899b0122b91a7
>>
>> I'm trying to debug some issues with spark and I'm having trouble
>> figuring out if this is part of the problem or if it's safe to ignore it.
>>
>> Any help or pointers would be appreciated.
>>
>> Thanks!
>> - Alan
>>
>
>

Re: Weird Spark Dispatcher Offers?

[Tim Chen]

Hi Alan,

The dispatcher is a Mesos framework and all frameworks in Mesos receives
offers from the master. Mesos is different than most schedulers where
we don't issue containers based on requests, but we offer available
resources to all frameworks and they in turn decide if they want to use
these resources.

In the Mesos dispatcher case we just decline offers coming in so it's
available for other frameworks.

Tim

On Fri, Oct 2, 2015 at 11:51 AM, Alan Braithwaite 
wrote:

> So if there is no jobs to run the dispatcher will decline all offers by
>> default.
>>
>
> So would this be a bug in mesos then?  I'm not sure I understand how this
> offer is appearing in the first place.  It only shows up in the master logs
> when I start the dispatcher.
>
>
>> Also we list all the jobs enqueued and it's specifications in the Spark
>> dispatcher UI, you should see the port in the dispatcher logs itself.
>
>
> Yes, this job is not listed under that UI.  Hence my confusion.
>
> Thanks,
> - Alan
>
> On Fri, Oct 2, 2015 at 11:49 AM, Tim Chen  wrote:
>
>> So if there is no jobs to run the dispatcher will decline all offers by
>> default.
>>
>> Also we list all the jobs enqueued and it's specifications in the Spark
>> dispatcher UI, you should see the port in the dispatcher logs itself.
>>
>> Tim
>>
>> On Fri, Oct 2, 2015 at 11:46 AM, Alan Braithwaite 
>> wrote:
>>
>>> This happened right after blowing away /var/lib/mesos zk://mesos and
>>> zk://spark_mesos_dispatcher and before I've submitted anything new to it so
>>> I _shouldn't_ have anything enqueued.  Unless there's state being stored
>>> somewhere besides those places that I don't know about.
>>>
>>> I'm not sure what the resource specifications are for this one because I
>>> didn't submit it directly.  If you have a way for me to grab a specific
>>> offer configuration, I'd be delighted to provide it.  I just can't seem to
>>> figure out how to get that information after digging through the mesos docs
>>> :-(
>>>
>>> Also, I can't read the docker logs because:
>>>
>>> Oct 02 11:39:59 sparky docker[556]:
>>> time="2015-10-02T11:39:59.165474049-07:00" level=error msg="Error streaming
>>> logs: invalid character '\\x00' looking for beginning of value"
>>>
>>> (that's coming from the spark-dispatcher docker).
>>>
>>> Thanks!
>>> - Alan
>>>
>>> On Fri, Oct 2, 2015 at 11:36 AM, Tim Chen  wrote:
>>>
 Do you have jobs enqueued? And if none of the jobs matches any offer it
 will just decline it.

 What's your job resource specifications?

 Tim

 On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite 
 wrote:

> Hey All,
>
> Using spark with mesos and docker.
>
> I'm wondering if anybody's seen the behavior of spark dispatcher where
> it just continually requests resources and immediately declines the offer.
>
> https://gist.github.com/anonymous/41e7c91899b0122b91a7
>
> I'm trying to debug some issues with spark and I'm having trouble
> figuring out if this is part of the problem or if it's safe to ignore it.
>
> Any help or pointers would be appreciated.
>
> Thanks!
> - Alan
>


>>>
>>
>

Re: Weird Spark Dispatcher Offers?

[Tim Chen]

So if there is no jobs to run the dispatcher will decline all offers by
default.

Also we list all the jobs enqueued and it's specifications in the Spark
dispatcher UI, you should see the port in the dispatcher logs itself.

Tim

On Fri, Oct 2, 2015 at 11:46 AM, Alan Braithwaite 
wrote:

> This happened right after blowing away /var/lib/mesos zk://mesos and
> zk://spark_mesos_dispatcher and before I've submitted anything new to it so
> I _shouldn't_ have anything enqueued.  Unless there's state being stored
> somewhere besides those places that I don't know about.
>
> I'm not sure what the resource specifications are for this one because I
> didn't submit it directly.  If you have a way for me to grab a specific
> offer configuration, I'd be delighted to provide it.  I just can't seem to
> figure out how to get that information after digging through the mesos docs
> :-(
>
> Also, I can't read the docker logs because:
>
> Oct 02 11:39:59 sparky docker[556]:
> time="2015-10-02T11:39:59.165474049-07:00" level=error msg="Error streaming
> logs: invalid character '\\x00' looking for beginning of value"
>
> (that's coming from the spark-dispatcher docker).
>
> Thanks!
> - Alan
>
> On Fri, Oct 2, 2015 at 11:36 AM, Tim Chen  wrote:
>
>> Do you have jobs enqueued? And if none of the jobs matches any offer it
>> will just decline it.
>>
>> What's your job resource specifications?
>>
>> Tim
>>
>> On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite 
>> wrote:
>>
>>> Hey All,
>>>
>>> Using spark with mesos and docker.
>>>
>>> I'm wondering if anybody's seen the behavior of spark dispatcher where
>>> it just continually requests resources and immediately declines the offer.
>>>
>>> https://gist.github.com/anonymous/41e7c91899b0122b91a7
>>>
>>> I'm trying to debug some issues with spark and I'm having trouble
>>> figuring out if this is part of the problem or if it's safe to ignore it.
>>>
>>> Any help or pointers would be appreciated.
>>>
>>> Thanks!
>>> - Alan
>>>
>>
>>
>