Hi Tim, Any way I can provide more info on this?
On Thu, Oct 1, 2015 at 4:21 PM, Utkarsh Sengar <utkarsh2...@gmail.com> wrote: > Not sure what you mean by that, I shared the data which I see in spark UI. > Can you point me to a location where I can precisely get the data you need? > > When I run the job in fine grained mode, I see tons are tasks created and > destroyed under a mesos "framework". I have about 80k spark tasks which I > think translates directly to independent mesos tasks. > > https://dl.dropboxusercontent.com/u/2432670/Screen%20Shot%202015-10-01%20at%204.14.34%20PM.png > > When i run the job in coarse grained mode, I just see 1-4 tasks with 1-4 > executors (it varies from what mesos allocates). And these mesos tasks try > to complete the 80k spark tasks and runs out of memory eventually (see the > stack track above) in the gist shared above. > > > On Thu, Oct 1, 2015 at 4:07 PM, Tim Chen <t...@mesosphere.io> wrote: > >> Hi Utkarsh, >> >> I replied earlier asking what is your task assignment like with fine vs >> coarse grain mode look like? >> >> Tim >> >> On Thu, Oct 1, 2015 at 4:05 PM, Utkarsh Sengar <utkarsh2...@gmail.com> >> wrote: >> >>> Bumping it up, its not really a blocking issue. >>> But fine grain mode eats up uncertain number of resources in mesos and >>> launches tons of tasks, so I would prefer using the coarse grained mode if >>> only it didn't run out of memory. >>> >>> Thanks, >>> -Utkarsh >>> >>> On Mon, Sep 28, 2015 at 2:24 PM, Utkarsh Sengar <utkarsh2...@gmail.com> >>> wrote: >>> >>>> Hi Tim, >>>> >>>> 1. spark.mesos.coarse:false (fine grain mode) >>>> This is the data dump for config and executors assigned: >>>> https://gist.github.com/utkarsh2012/6401d5526feccab14687 >>>> >>>> 2. spark.mesos.coarse:true (coarse grain mode) >>>> Dump for coarse mode: >>>> https://gist.github.com/utkarsh2012/918cf6f8ed5945627188 >>>> >>>> As you can see, exactly the same code works fine in fine grained, goes >>>> out of memory in coarse grained mode. First an executor was lost and then >>>> the driver went out of memory. >>>> So I am trying to understand what is different in fine grained vs >>>> coarse mode other than allocation of multiple mesos tasks vs 1 mesos task. >>>> Clearly spark is not managing memory in the same way. >>>> >>>> Thanks, >>>> -Utkarsh >>>> >>>> >>>> On Fri, Sep 25, 2015 at 9:17 AM, Tim Chen <t...@mesosphere.io> wrote: >>>> >>>>> Hi Utkarsh, >>>>> >>>>> What is your job placement like when you run fine grain mode? You said >>>>> coarse grain mode only ran with one node right? >>>>> >>>>> And when the job is running could you open the Spark webui and get >>>>> stats about the heap size and other java settings? >>>>> >>>>> Tim >>>>> >>>>> On Thu, Sep 24, 2015 at 10:56 PM, Utkarsh Sengar < >>>>> utkarsh2...@gmail.com> wrote: >>>>> >>>>>> Bumping this one up, any suggestions on the stacktrace? >>>>>> spark.mesos.coarse=true is not working and the driver crashed with >>>>>> the error. >>>>>> >>>>>> On Wed, Sep 23, 2015 at 3:29 PM, Utkarsh Sengar < >>>>>> utkarsh2...@gmail.com> wrote: >>>>>> >>>>>>> Missed to do a reply-all. >>>>>>> >>>>>>> Tim, >>>>>>> >>>>>>> spark.mesos.coarse = true doesn't work and spark.mesos.coarse = >>>>>>> false works (sorry there was a typo in my last email, I meant "when I do >>>>>>> "spark.mesos.coarse=false", the job works like a charm. "). >>>>>>> >>>>>>> I get this exception with spark.mesos.coarse = true: >>>>>>> >>>>>>> 15/09/22 20:18:05 INFO MongoCollectionSplitter: Created split: min={ >>>>>>> "_id" : "55af4bf26750ad38a444d7cf"}, max= { "_id" : >>>>>>> "55af5a61e8a42806f47546c1"} >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611337>15/09/22 >>>>>>> 20:18:05 INFO MongoCollectionSplitter: Created split: min={ "_id" : >>>>>>> "55af5a61e8a42806f47546c1"}, max= null >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611453>Exception >>>>>>> in thread "main" java.lang.OutOfMemoryError: Java heap space >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611524> >>>>>>> at >>>>>>> org.apache.spark.rdd.CartesianRDD.getPartitions(CartesianRDD.scala:60) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611599> >>>>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611671> >>>>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611743> >>>>>>> at scala.Option.getOrElse(Option.scala:120) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611788> >>>>>>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611843> >>>>>>> at >>>>>>> org.apache.spark.rdd.CartesianRDD.getPartitions(CartesianRDD.scala:60) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611918> >>>>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611990> >>>>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612062> >>>>>>> at scala.Option.getOrElse(Option.scala:120) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612107> >>>>>>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612162> >>>>>>> at >>>>>>> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612245> >>>>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612317> >>>>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612389> >>>>>>> at scala.Option.getOrElse(Option.scala:120) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612434> >>>>>>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612489> >>>>>>> at >>>>>>> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612572> >>>>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612644> >>>>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612716> >>>>>>> at scala.Option.getOrElse(Option.scala:120) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612761> >>>>>>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612816> >>>>>>> at >>>>>>> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612899> >>>>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612971> >>>>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613043> >>>>>>> at scala.Option.getOrElse(Option.scala:120) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613088> >>>>>>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613143> >>>>>>> at >>>>>>> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613226> >>>>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613298> >>>>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613370> >>>>>>> at scala.Option.getOrElse(Option.scala:120) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613415> >>>>>>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613470> >>>>>>> at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:82) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613537> >>>>>>> at >>>>>>> org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:78) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613612>15/09/22 >>>>>>> 20:18:17 INFO SparkContext: Invoking stop() from shutdown hook >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613684>15/09/22 >>>>>>> 20:18:17 INFO BlockManagerInfo: Removed broadcast_2_piece0 on >>>>>>> some-ip-here:37706 in memory (size: 1964.0 B, free: 2.8 GB) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613814>15/09/22 >>>>>>> 20:18:17 INFO BlockManagerInfo: Removed broadcast_2_piece0 on >>>>>>> mesos-slave10 >>>>>>> in memory (size: 1964.0 B, free: 5.2 GB) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613977>15/09/22 >>>>>>> 20:18:17 INFO BlockManagerInfo: Removed broadcast_1_piece0 on >>>>>>> some-ip-here:37706 in memory (size: 17.2 KB, free: 2.8 GB) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614106>15/09/22 >>>>>>> 20:18:17 INFO BlockManagerInfo: Removed broadcast_1_piece0 on >>>>>>> mesos-slave105 in memory (size: 17.2 KB, free: 5.2 GB) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614268>15/09/22 >>>>>>> 20:18:17 INFO BlockManagerInfo: Removed broadcast_1_piece0 on >>>>>>> mesos-slave1 >>>>>>> in memory (size: 17.2 KB, free: 5.2 GB) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614429>15/09/22 >>>>>>> 20:18:17 INFO BlockManagerInfo: Removed broadcast_1_piece0 on >>>>>>> mesos-slave9 >>>>>>> in memory (size: 17.2 KB, free: 5.2 GB) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614590>15/09/22 >>>>>>> 20:18:17 INFO BlockManagerInfo: Removed broadcast_1_piece0 on >>>>>>> mesos-slave3 >>>>>>> in memory (size: 17.2 KB, free: 5.2 GB) >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614751>15/09/22 >>>>>>> 20:18:17 INFO SparkUI: Stopped Spark web UI at >>>>>>> http://some-ip-here:4040 >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614831>15/09/22 >>>>>>> 20:18:17 INFO DAGScheduler: Stopping DAGScheduler >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614890>15/09/22 >>>>>>> 20:18:17 INFO CoarseMesosSchedulerBackend: Shutting down all executors >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614970>15/09/22 >>>>>>> 20:18:17 INFO CoarseMesosSchedulerBackend: Asking each executor to shut >>>>>>> down >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615056>I0922 >>>>>>> 20:18:17.794598 171 sched.cpp:1591] Asked to stop the driver >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615125>I0922 >>>>>>> 20:18:17.794739 143 sched.cpp:835] Stopping framework >>>>>>> '20150803-224832-1577534986-5050-1614-0016' >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615231>15/09/22 >>>>>>> 20:18:17 INFO CoarseMesosSchedulerBackend: driver.run() returned with >>>>>>> code >>>>>>> DRIVER_STOPPED >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615330>15/09/22 >>>>>>> 20:18:17 INFO MapOutputTrackerMasterEndpoint: >>>>>>> MapOutputTrackerMasterEndpoint stopped! >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615425>15/09/22 >>>>>>> 20:18:17 INFO Utils: path = >>>>>>> /tmp/spark-98801318-9c49-473b-bf2f-07ea42187252/blockmgr-0e0e1a1c-894e-4e79-beac-ead0dff43166, >>>>>>> already present as root for deletion. >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615595>15/09/22 >>>>>>> 20:18:17 INFO MemoryStore: MemoryStore cleared >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615651>15/09/22 >>>>>>> 20:18:17 INFO BlockManager: BlockManager stopped >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615709>15/09/22 >>>>>>> 20:18:17 INFO BlockManagerMaster: BlockManagerMaster stopped >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615779>15/09/22 >>>>>>> 20:18:17 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: >>>>>>> OutputCommitCoordinator stopped! >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615892>15/09/22 >>>>>>> 20:18:17 INFO SparkContext: Successfully stopped SparkContext >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615963>15/09/22 >>>>>>> 20:18:17 INFO Utils: Shutdown hook called >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#616014>15/09/22 >>>>>>> 20:18:17 INFO Utils: Deleting directory >>>>>>> /tmp/spark-98801318-9c49-473b-bf2f-07ea42187252 >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#616111>15/09/22 >>>>>>> 20:18:17 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down >>>>>>> remote daemon. >>>>>>> >>>>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#616206>15/09/22 >>>>>>> 20:18:17 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon >>>>>>> shut >>>>>>> down; proceeding with flushing remote transports. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Sep 22, 2015 at 1:26 AM, Tim Chen <t...@mesosphere.io> wrote: >>>>>>> >>>>>>>> Hi Utkarsh, >>>>>>>> >>>>>>>> Just to be sure you originally set coarse to false but then to >>>>>>>> true? Or is it the other way around? >>>>>>>> >>>>>>>> Also what's the exception/stack trace when the driver crashed? >>>>>>>> >>>>>>>> Coarse grain mode per-starts all the Spark executor backends, so >>>>>>>> has the least overhead comparing to fine grain. There is no single >>>>>>>> answer >>>>>>>> for which mode you should use, otherwise we would have removed one of >>>>>>>> those >>>>>>>> modes since it depends on your use case. >>>>>>>> >>>>>>>> There are quite some factor why there could be huge GC pauses, but >>>>>>>> I don't think if you switch to standalone your GC pauses go away. >>>>>>>> >>>>>>>> Tim >>>>>>>> >>>>>>>> On Mon, Sep 21, 2015 at 5:18 PM, Utkarsh Sengar < >>>>>>>> utkarsh2...@gmail.com> wrote: >>>>>>>> >>>>>>>>> I am running Spark 1.4.1 on mesos. >>>>>>>>> >>>>>>>>> The spark job does a "cartesian" of 4 RDDs (aRdd, bRdd, cRdd, >>>>>>>>> dRdd) of size 100, 100, 7 and 1 respectively. Lets call it prouctRDD. >>>>>>>>> >>>>>>>>> Creation of "aRdd" needs data pull from multiple data sources, >>>>>>>>> merging it and creating a tuple of JavaRdd, finally aRDD looks >>>>>>>>> something >>>>>>>>> like this: JavaRDD<Tuple4<A1, A2>> >>>>>>>>> bRdd, cRdd and dRdds are just List<> of values. >>>>>>>>> >>>>>>>>> Then apply a transformation on prouctRDD and finally call >>>>>>>>> "saveAsTextFile" to save the result of my transformation. >>>>>>>>> >>>>>>>>> Problem: >>>>>>>>> By setting "spark.mesos.coarse=true", creation of "aRdd" works >>>>>>>>> fine but driver crashes while doing the cartesian but when I do >>>>>>>>> "spark.mesos.coarse=true", the job works like a charm. I am running >>>>>>>>> spark >>>>>>>>> on mesos. >>>>>>>>> >>>>>>>>> Comments: >>>>>>>>> So I wanted to understand what role does "spark.mesos.coarse=true" >>>>>>>>> plays in terms of memory and compute performance. My findings look >>>>>>>>> counter >>>>>>>>> intuitive since: >>>>>>>>> >>>>>>>>> 1. "spark.mesos.coarse=true" just runs on 1 mesos task, so >>>>>>>>> there should be an overhead of spinning up mesos tasks which >>>>>>>>> should impact >>>>>>>>> the performance. >>>>>>>>> 2. What config for "spark.mesos.coarse" recommended for >>>>>>>>> running spark on mesos? Or there is no best answer and it depends >>>>>>>>> on >>>>>>>>> usecase? >>>>>>>>> 3. Also by setting "spark.mesos.coarse=true", I notice that I >>>>>>>>> get huge GC pauses even with small dataset but a long running job >>>>>>>>> (but this >>>>>>>>> can be a separate discussion). >>>>>>>>> >>>>>>>>> Let me know if I am missing something obvious, we are learning >>>>>>>>> spark tuning as we move forward :) >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thanks, >>>>>>>>> -Utkarsh >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks, >>>>>>> -Utkarsh >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks, >>>>>> -Utkarsh >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks, >>>> -Utkarsh >>>> >>> >>> >>> >>> -- >>> Thanks, >>> -Utkarsh >>> >> >> > > > -- > Thanks, > -Utkarsh > -- Thanks, -Utkarsh
