Re: Weird Spark Dispatcher Offers?
Hi Alan, The dispatcher is a Mesos framework and all frameworks in Mesos receives offers from the master. Mesos is different than most schedulers where we don't issue containers based on requests, but we offer available resources to all frameworks and they in turn decide if they want to use these resources. In the Mesos dispatcher case we just decline offers coming in so it's available for other frameworks. Tim On Fri, Oct 2, 2015 at 11:51 AM, Alan Braithwaite wrote: > So if there is no jobs to run the dispatcher will decline all offers by >> default. >> > > So would this be a bug in mesos then? I'm not sure I understand how this > offer is appearing in the first place. It only shows up in the master logs > when I start the dispatcher. > > >> Also we list all the jobs enqueued and it's specifications in the Spark >> dispatcher UI, you should see the port in the dispatcher logs itself. > > > Yes, this job is not listed under that UI. Hence my confusion. > > Thanks, > - Alan > > On Fri, Oct 2, 2015 at 11:49 AM, Tim Chen wrote: > >> So if there is no jobs to run the dispatcher will decline all offers by >> default. >> >> Also we list all the jobs enqueued and it's specifications in the Spark >> dispatcher UI, you should see the port in the dispatcher logs itself. >> >> Tim >> >> On Fri, Oct 2, 2015 at 11:46 AM, Alan Braithwaite >> wrote: >> >>> This happened right after blowing away /var/lib/mesos zk://mesos and >>> zk://spark_mesos_dispatcher and before I've submitted anything new to it so >>> I _shouldn't_ have anything enqueued. Unless there's state being stored >>> somewhere besides those places that I don't know about. >>> >>> I'm not sure what the resource specifications are for this one because I >>> didn't submit it directly. If you have a way for me to grab a specific >>> offer configuration, I'd be delighted to provide it. I just can't seem to >>> figure out how to get that information after digging through the mesos docs >>> :-( >>> >>> Also, I can't read the docker logs because: >>> >>> Oct 02 11:39:59 sparky docker[556]: >>> time="2015-10-02T11:39:59.165474049-07:00" level=error msg="Error streaming >>> logs: invalid character '\\x00' looking for beginning of value" >>> >>> (that's coming from the spark-dispatcher docker). >>> >>> Thanks! >>> - Alan >>> >>> On Fri, Oct 2, 2015 at 11:36 AM, Tim Chen wrote: >>> Do you have jobs enqueued? And if none of the jobs matches any offer it will just decline it. What's your job resource specifications? Tim On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite wrote: > Hey All, > > Using spark with mesos and docker. > > I'm wondering if anybody's seen the behavior of spark dispatcher where > it just continually requests resources and immediately declines the offer. > > https://gist.github.com/anonymous/41e7c91899b0122b91a7 > > I'm trying to debug some issues with spark and I'm having trouble > figuring out if this is part of the problem or if it's safe to ignore it. > > Any help or pointers would be appreciated. > > Thanks! > - Alan > >>> >> >
Re: Weird Spark Dispatcher Offers?
> > So if there is no jobs to run the dispatcher will decline all offers by > default. > So would this be a bug in mesos then? I'm not sure I understand how this offer is appearing in the first place. It only shows up in the master logs when I start the dispatcher. > Also we list all the jobs enqueued and it's specifications in the Spark > dispatcher UI, you should see the port in the dispatcher logs itself. Yes, this job is not listed under that UI. Hence my confusion. Thanks, - Alan On Fri, Oct 2, 2015 at 11:49 AM, Tim Chen wrote: > So if there is no jobs to run the dispatcher will decline all offers by > default. > > Also we list all the jobs enqueued and it's specifications in the Spark > dispatcher UI, you should see the port in the dispatcher logs itself. > > Tim > > On Fri, Oct 2, 2015 at 11:46 AM, Alan Braithwaite > wrote: > >> This happened right after blowing away /var/lib/mesos zk://mesos and >> zk://spark_mesos_dispatcher and before I've submitted anything new to it so >> I _shouldn't_ have anything enqueued. Unless there's state being stored >> somewhere besides those places that I don't know about. >> >> I'm not sure what the resource specifications are for this one because I >> didn't submit it directly. If you have a way for me to grab a specific >> offer configuration, I'd be delighted to provide it. I just can't seem to >> figure out how to get that information after digging through the mesos docs >> :-( >> >> Also, I can't read the docker logs because: >> >> Oct 02 11:39:59 sparky docker[556]: >> time="2015-10-02T11:39:59.165474049-07:00" level=error msg="Error streaming >> logs: invalid character '\\x00' looking for beginning of value" >> >> (that's coming from the spark-dispatcher docker). >> >> Thanks! >> - Alan >> >> On Fri, Oct 2, 2015 at 11:36 AM, Tim Chen wrote: >> >>> Do you have jobs enqueued? And if none of the jobs matches any offer it >>> will just decline it. >>> >>> What's your job resource specifications? >>> >>> Tim >>> >>> On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite >>> wrote: >>> Hey All, Using spark with mesos and docker. I'm wondering if anybody's seen the behavior of spark dispatcher where it just continually requests resources and immediately declines the offer. https://gist.github.com/anonymous/41e7c91899b0122b91a7 I'm trying to debug some issues with spark and I'm having trouble figuring out if this is part of the problem or if it's safe to ignore it. Any help or pointers would be appreciated. Thanks! - Alan >>> >>> >> >
Re: Weird Spark Dispatcher Offers?
So if there is no jobs to run the dispatcher will decline all offers by default. Also we list all the jobs enqueued and it's specifications in the Spark dispatcher UI, you should see the port in the dispatcher logs itself. Tim On Fri, Oct 2, 2015 at 11:46 AM, Alan Braithwaite wrote: > This happened right after blowing away /var/lib/mesos zk://mesos and > zk://spark_mesos_dispatcher and before I've submitted anything new to it so > I _shouldn't_ have anything enqueued. Unless there's state being stored > somewhere besides those places that I don't know about. > > I'm not sure what the resource specifications are for this one because I > didn't submit it directly. If you have a way for me to grab a specific > offer configuration, I'd be delighted to provide it. I just can't seem to > figure out how to get that information after digging through the mesos docs > :-( > > Also, I can't read the docker logs because: > > Oct 02 11:39:59 sparky docker[556]: > time="2015-10-02T11:39:59.165474049-07:00" level=error msg="Error streaming > logs: invalid character '\\x00' looking for beginning of value" > > (that's coming from the spark-dispatcher docker). > > Thanks! > - Alan > > On Fri, Oct 2, 2015 at 11:36 AM, Tim Chen wrote: > >> Do you have jobs enqueued? And if none of the jobs matches any offer it >> will just decline it. >> >> What's your job resource specifications? >> >> Tim >> >> On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite >> wrote: >> >>> Hey All, >>> >>> Using spark with mesos and docker. >>> >>> I'm wondering if anybody's seen the behavior of spark dispatcher where >>> it just continually requests resources and immediately declines the offer. >>> >>> https://gist.github.com/anonymous/41e7c91899b0122b91a7 >>> >>> I'm trying to debug some issues with spark and I'm having trouble >>> figuring out if this is part of the problem or if it's safe to ignore it. >>> >>> Any help or pointers would be appreciated. >>> >>> Thanks! >>> - Alan >>> >> >> >
Re: Weird Spark Dispatcher Offers?
This happened right after blowing away /var/lib/mesos zk://mesos and zk://spark_mesos_dispatcher and before I've submitted anything new to it so I _shouldn't_ have anything enqueued. Unless there's state being stored somewhere besides those places that I don't know about. I'm not sure what the resource specifications are for this one because I didn't submit it directly. If you have a way for me to grab a specific offer configuration, I'd be delighted to provide it. I just can't seem to figure out how to get that information after digging through the mesos docs :-( Also, I can't read the docker logs because: Oct 02 11:39:59 sparky docker[556]: time="2015-10-02T11:39:59.165474049-07:00" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value" (that's coming from the spark-dispatcher docker). Thanks! - Alan On Fri, Oct 2, 2015 at 11:36 AM, Tim Chen wrote: > Do you have jobs enqueued? And if none of the jobs matches any offer it > will just decline it. > > What's your job resource specifications? > > Tim > > On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite > wrote: > >> Hey All, >> >> Using spark with mesos and docker. >> >> I'm wondering if anybody's seen the behavior of spark dispatcher where it >> just continually requests resources and immediately declines the offer. >> >> https://gist.github.com/anonymous/41e7c91899b0122b91a7 >> >> I'm trying to debug some issues with spark and I'm having trouble >> figuring out if this is part of the problem or if it's safe to ignore it. >> >> Any help or pointers would be appreciated. >> >> Thanks! >> - Alan >> > >
Re: Weird Spark Dispatcher Offers?
Do you have jobs enqueued? And if none of the jobs matches any offer it will just decline it. What's your job resource specifications? Tim On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite wrote: > Hey All, > > Using spark with mesos and docker. > > I'm wondering if anybody's seen the behavior of spark dispatcher where it > just continually requests resources and immediately declines the offer. > > https://gist.github.com/anonymous/41e7c91899b0122b91a7 > > I'm trying to debug some issues with spark and I'm having trouble figuring > out if this is part of the problem or if it's safe to ignore it. > > Any help or pointers would be appreciated. > > Thanks! > - Alan >
Weird Spark Dispatcher Offers?
Hey All, Using spark with mesos and docker. I'm wondering if anybody's seen the behavior of spark dispatcher where it just continually requests resources and immediately declines the offer. https://gist.github.com/anonymous/41e7c91899b0122b91a7 I'm trying to debug some issues with spark and I'm having trouble figuring out if this is part of the problem or if it's safe to ignore it. Any help or pointers would be appreciated. Thanks! - Alan