On Thu, Sep 10, 2015 at 3:35 AM, canan chen <ccn...@gmail.com> wrote:
> Finally got the answer. Actually it works fine. The allocation behavior > on mesos is a little different from yarn/standalone. Seems the executor in > mesos is lazily allocated (only when job is executed) while executor in > yarn/standalone is allocated when spark-shell is started. > That's in fine-grained mode (the default). You can turn on coarse-grained mode to acquire executors on startup. iulian > > > > On Tue, Sep 8, 2015 at 10:39 PM, canan chen <ccn...@gmail.com> wrote: > >> Yes, I follow the guide in this doc, and run it as mesos client mode >> >> On Tue, Sep 8, 2015 at 6:31 PM, Akhil Das <ak...@sigmoidanalytics.com> >> wrote: >> >>> In which mode are you submitting your application? (coarse-grained or >>> fine-grained(default)). Have you gone through this documentation already? >>> http://spark.apache.org/docs/latest/running-on-mesos.html#using-a-mesos-master-url >>> >>> Thanks >>> Best Regards >>> >>> On Tue, Sep 8, 2015 at 12:54 PM, canan chen <ccn...@gmail.com> wrote: >>> >>>> Hi all, >>>> >>>> I try to run spark on mesos, but it looks like I can not allocate >>>> resources from mesos. I am not expert of mesos, but from the mesos log, it >>>> seems spark always decline the offer from mesos. Not sure what's wrong, >>>> maybe need some configuration change. Here's the mesos master log >>>> >>>> I0908 15:08:16.515960 301916160 master.cpp:1767] Received registration >>>> request for framework 'Spark shell' at >>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 >>>> I0908 15:08:16.520545 301916160 master.cpp:1834] Registering framework >>>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at >>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 with >>>> checkpointing disabled and capabilities [ ] >>>> I0908 15:08:16.522307 300843008 hierarchical.hpp:386] Added framework >>>> 20150908-143320-16777343-5050-41965-0000 >>>> I0908 15:08:16.525845 301379584 master.cpp:4290] Sending 1 offers to >>>> framework 20150908-143320-16777343-5050-41965-0000 (Spark shell) at >>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 >>>> I0908 15:08:16.637677 302452736 master.cpp:2884] Processing DECLINE >>>> call for offers: [ 20150908-143320-16777343-5050-41965-O0 ] for framework >>>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at >>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 >>>> I0908 15:08:16.639197 299233280 hierarchical.hpp:761] Recovered >>>> cpus(*):8; mem(*):15360; disk(*):470842; ports(*):[31000-32000] (total: >>>> cpus(*):8; mem(*):15360; disk(*):470842; ports(*):[31000-32000], allocated: >>>> ) on slave 20150908-143320-16777343-5050-41965-S0 from framework >>>> 20150908-143320-16777343-5050-41965-0000 >>>> I0908 15:08:21.786932 300306432 master.cpp:4290] Sending 1 offers to >>>> framework 20150908-143320-16777343-5050-41965-0000 (Spark shell) at >>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 >>>> I0908 15:08:21.789979 298696704 master.cpp:2884] Processing DECLINE >>>> call for offers: [ 20150908-143320-16777343-5050-41965-O1 ] for framework >>>> 20150908-143320-16777343-5050-41965-0000 (Spark shell) at >>>> scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 >>>> >>> >>> >> > -- -- Iulian Dragos ------ Reactive Apps on the JVM www.typesafe.com
