Thank you Silvio.
I have not actually tried dynamic allocation (it's not trivial to use an external shuffle service). I will do more research there. What we are actually experiencing is Mesos offer starvation caused by many long-lived Spark frameworks hoarding "Mesos resource offers" (not resources). This is why I am considering closing/restarting the Spark interpreter rather than using dynamic allocation or fine-grained mode to keep resource usage at a minimum. For example, looking through the current API documents, it seems possible for a "watchdog" process to determine if a note is running, when it has last ran, and then restart the interpreter. ________________________________ From: Silvio Fiorito <silvio.fior...@granturing.com> Sent: Friday, March 4, 2016 9:05 AM To: users@zeppelin.incubator.apache.org; Dylan Meissner Subject: Re: Spark interpreter idle timeout If you're using Mesos as your Spark cluster manager, then you can use dynamic resource allocation. So as your users are running notebooks the Spark executors will scale up and down as needed, per the thresholds you define. And when the user is idle, Spark will automatically release resources. Please see the docs here, for more info: http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation [http://spark.apache.org/docs/latest/img/spark-logo-hd.png]<http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation> Job Scheduling - Spark 1.6.0 Documentation<http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation> spark.apache.org Job Scheduling. Overview; Scheduling Across Applications. Dynamic Resource Allocation. Configuration and Setup; Resource Allocation Policy. Request Policy http://spark.apache.org/docs/latest/running-on-mesos.html#dynamic-resource-allocation-with-mesos Thanks, Silvio From: Dylan Meissner <dylan.meiss...@gettyimages.com<mailto:dylan.meiss...@gettyimages.com>> Reply-To: "users@zeppelin.incubator.apache.org<mailto:users@zeppelin.incubator.apache.org>" <users@zeppelin.incubator.apache.org<mailto:users@zeppelin.incubator.apache.org>> Date: Friday, March 4, 2016 at 11:52 AM To: "users@zeppelin.incubator.apache.org<mailto:users@zeppelin.incubator.apache.org>" <users@zeppelin.incubator.apache.org<mailto:users@zeppelin.incubator.apache.org>> Subject: Spark interpreter idle timeout Greetings, We run multiple Zeppelin's per user in a Mesos cluster. The Mesos Marathon framework hosts the Zeppelin servers, and running a note causes a Spark framework to start a Spark context to distribute the workload described in the notes. This works well for us. However, when notebooks are left unattended, we'd like the Spark interpreter to shut down. This will free resources that can go to other Mesos frameworks. Is there a way to set an "idle timeout" today, and if not, how do you imagine it could be accomplished in either Zeppelin, or Spark? Thanks, Dylan Meissner www.gettyimages.com<http://www.gettyimages.com>
