Understanding Mesos Maintenance

[Zameer Manji]

Hey,

I'm trying to understand some nuances of the maintenance API. Here are my
questions:

1. The documentation mentions that accepting or declining and inverse offer
is a "hint" to the operator. How do operators view if a framework has
declined, accepted or ignored an inverse offer?

2. Should a framework accept an inverse offer and then start removing tasks
from an agent or should the framework only accept the inverse offer after
the removal of tasks is complete? I think the former makes sense, but it
implies that operators need to poll the state of the agent to ensure there
are no active tasks whereas the latter implies operators only need to check
if all inverse offers were accepted.

3. After accepting the inverse offer, will a framework get another inverse
offer for the same agent? Currently I'm trying to determine if inverse
offer information needs to be persisted so a framework can continue it's
draining work between failovers or if it can just wait for an inverse offer
after starting up.

4. Is it possible for the agent to automatically transition from DRAIN to
DOWN if at the start of the unavailability period the agent is free of
tasks or is that still the operator's responsibility?

-- 
Zameer Manji

Re: Welcome Kevin Klues as a Mesos Committer and PMC member!

[Timothy Chen]

Congrats Kevin!

Tim

On Wed, Mar 1, 2017 at 3:20 PM, Neil Conway  wrote:
> Congratulations Kevin! Very well-deserved.
>
> Neil
>
> On Wed, Mar 1, 2017 at 2:05 PM, Benjamin Mahler  wrote:
>> Hi all,
>>
>> Please welcome Kevin Klues as the newest committer and PMC member of the
>> Apache Mesos project.
>>
>> Kevin has been an active contributor in the project for over a year, and in
>> this time he made a number of contributions to the project: Nvidia GPU
>> support [1], the containerization side of POD support (new container init
>> process), and support for "attach" and "exec" of commands within running
>> containers [2].
>>
>> Also, Kevin took on an effort with Haris Choudhary to revive the CLI [3]
>> via a better structured python implementation (to be more accessible to
>> contributors) and a more extensible architecture to better support adding
>> new or custom subcommands. The work also adds a unit test framework for the
>> CLI functionality (we had no tests previously!). I think it's great that
>> Kevin took on this much needed improvement with Haris, and I'm very much
>> looking forward to seeing this land in the project.
>>
>> Here is his committer eligibility document for perusal:
>> https://docs.google.com/document/d/1mlO1yyLCoCSd85XeDKIxTYyboK_uiOJ4Uwr6ruKTlFM/edit
>>
>> Thanks!
>> Ben
>>
>> [1] http://mesos.apache.org/documentation/latest/gpu-support/
>> [2]
>> https://docs.google.com/document/d/1nAVr0sSSpbDLrgUlAEB5hKzCl482NSVk8V0D56sFMzU
>> [3]
>> https://docs.google.com/document/d/1r6Iv4Efu8v8IBrcUTjgYkvZ32WVscgYqrD07OyIglsA/

Re: Welcome Kevin Klues as a Mesos Committer and PMC member!

[Neil Conway]

Congratulations Kevin! Very well-deserved.

Neil

On Wed, Mar 1, 2017 at 2:05 PM, Benjamin Mahler  wrote:
> Hi all,
>
> Please welcome Kevin Klues as the newest committer and PMC member of the
> Apache Mesos project.
>
> Kevin has been an active contributor in the project for over a year, and in
> this time he made a number of contributions to the project: Nvidia GPU
> support [1], the containerization side of POD support (new container init
> process), and support for "attach" and "exec" of commands within running
> containers [2].
>
> Also, Kevin took on an effort with Haris Choudhary to revive the CLI [3]
> via a better structured python implementation (to be more accessible to
> contributors) and a more extensible architecture to better support adding
> new or custom subcommands. The work also adds a unit test framework for the
> CLI functionality (we had no tests previously!). I think it's great that
> Kevin took on this much needed improvement with Haris, and I'm very much
> looking forward to seeing this land in the project.
>
> Here is his committer eligibility document for perusal:
> https://docs.google.com/document/d/1mlO1yyLCoCSd85XeDKIxTYyboK_uiOJ4Uwr6ruKTlFM/edit
>
> Thanks!
> Ben
>
> [1] http://mesos.apache.org/documentation/latest/gpu-support/
> [2]
> https://docs.google.com/document/d/1nAVr0sSSpbDLrgUlAEB5hKzCl482NSVk8V0D56sFMzU
> [3]
> https://docs.google.com/document/d/1r6Iv4Efu8v8IBrcUTjgYkvZ32WVscgYqrD07OyIglsA/

Re: Welcome Kevin Klues as a Mesos Committer and PMC member!

[Qian Zhang]

Congratulations!


Thanks,
Qian Zhang

On Thu, Mar 2, 2017 at 6:43 AM, Greg Mann  wrote:

> Woowoo! Congrats Kevin!!
>
> On Wed, Mar 1, 2017 at 2:26 PM, Avinash Sridharan 
> wrote:
>
>> Awesome !! Congrats Kevin !!
>>
>> On Wed, Mar 1, 2017 at 2:07 PM, Jie Yu  wrote:
>>
>>> Congrats! Kevin! Well deserved!
>>>
>>> On Wed, Mar 1, 2017 at 2:05 PM, Benjamin Mahler 
>>> wrote:
>>>
>>> > Hi all,
>>> >
>>> > Please welcome Kevin Klues as the newest committer and PMC member of
>>> the
>>> > Apache Mesos project.
>>> >
>>> > Kevin has been an active contributor in the project for over a year,
>>> and in
>>> > this time he made a number of contributions to the project: Nvidia GPU
>>> > support [1], the containerization side of POD support (new container
>>> init
>>> > process), and support for "attach" and "exec" of commands within
>>> running
>>> > containers [2].
>>> >
>>> > Also, Kevin took on an effort with Haris Choudhary to revive the CLI
>>> [3]
>>> > via a better structured python implementation (to be more accessible to
>>> > contributors) and a more extensible architecture to better support
>>> adding
>>> > new or custom subcommands. The work also adds a unit test framework
>>> for the
>>> > CLI functionality (we had no tests previously!). I think it's great
>>> that
>>> > Kevin took on this much needed improvement with Haris, and I'm very
>>> much
>>> > looking forward to seeing this land in the project.
>>> >
>>> > Here is his committer eligibility document for perusal:
>>> > https://docs.google.com/document/d/1mlO1yyLCoCSd85XeDKIxTYyboK_
>>> > uiOJ4Uwr6ruKTlFM/edit
>>> >
>>> > Thanks!
>>> > Ben
>>> >
>>> > [1] http://mesos.apache.org/documentation/latest/gpu-support/
>>> > [2]
>>> > https://docs.google.com/document/d/1nAVr0sSSpbDLrgUlAEB5hKzCl482N
>>> > SVk8V0D56sFMzU
>>> > [3]
>>> > https://docs.google.com/document/d/1r6Iv4Efu8v8IBrcUTjgYkvZ32WVsc
>>> > gYqrD07OyIglsA/
>>> >
>>>
>>
>>
>>
>> --
>> Avinash Sridharan, Mesosphere
>> +1 (323) 702 5245 <(323)%20702-5245>
>>
>
>

Re: Welcome Kevin Klues as a Mesos Committer and PMC member!

[Greg Mann]

Woowoo! Congrats Kevin!!

On Wed, Mar 1, 2017 at 2:26 PM, Avinash Sridharan 
wrote:

> Awesome !! Congrats Kevin !!
>
> On Wed, Mar 1, 2017 at 2:07 PM, Jie Yu  wrote:
>
>> Congrats! Kevin! Well deserved!
>>
>> On Wed, Mar 1, 2017 at 2:05 PM, Benjamin Mahler 
>> wrote:
>>
>> > Hi all,
>> >
>> > Please welcome Kevin Klues as the newest committer and PMC member of the
>> > Apache Mesos project.
>> >
>> > Kevin has been an active contributor in the project for over a year,
>> and in
>> > this time he made a number of contributions to the project: Nvidia GPU
>> > support [1], the containerization side of POD support (new container
>> init
>> > process), and support for "attach" and "exec" of commands within running
>> > containers [2].
>> >
>> > Also, Kevin took on an effort with Haris Choudhary to revive the CLI [3]
>> > via a better structured python implementation (to be more accessible to
>> > contributors) and a more extensible architecture to better support
>> adding
>> > new or custom subcommands. The work also adds a unit test framework for
>> the
>> > CLI functionality (we had no tests previously!). I think it's great that
>> > Kevin took on this much needed improvement with Haris, and I'm very much
>> > looking forward to seeing this land in the project.
>> >
>> > Here is his committer eligibility document for perusal:
>> > https://docs.google.com/document/d/1mlO1yyLCoCSd85XeDKIxTYyboK_
>> > uiOJ4Uwr6ruKTlFM/edit
>> >
>> > Thanks!
>> > Ben
>> >
>> > [1] http://mesos.apache.org/documentation/latest/gpu-support/
>> > [2]
>> > https://docs.google.com/document/d/1nAVr0sSSpbDLrgUlAEB5hKzCl482N
>> > SVk8V0D56sFMzU
>> > [3]
>> > https://docs.google.com/document/d/1r6Iv4Efu8v8IBrcUTjgYkvZ32WVsc
>> > gYqrD07OyIglsA/
>> >
>>
>
>
>
> --
> Avinash Sridharan, Mesosphere
> +1 (323) 702 5245 <(323)%20702-5245>
>

Re: Welcome Kevin Klues as a Mesos Committer and PMC member!

[Avinash Sridharan]

Awesome !! Congrats Kevin !!

On Wed, Mar 1, 2017 at 2:07 PM, Jie Yu  wrote:

> Congrats! Kevin! Well deserved!
>
> On Wed, Mar 1, 2017 at 2:05 PM, Benjamin Mahler 
> wrote:
>
> > Hi all,
> >
> > Please welcome Kevin Klues as the newest committer and PMC member of the
> > Apache Mesos project.
> >
> > Kevin has been an active contributor in the project for over a year, and
> in
> > this time he made a number of contributions to the project: Nvidia GPU
> > support [1], the containerization side of POD support (new container init
> > process), and support for "attach" and "exec" of commands within running
> > containers [2].
> >
> > Also, Kevin took on an effort with Haris Choudhary to revive the CLI [3]
> > via a better structured python implementation (to be more accessible to
> > contributors) and a more extensible architecture to better support adding
> > new or custom subcommands. The work also adds a unit test framework for
> the
> > CLI functionality (we had no tests previously!). I think it's great that
> > Kevin took on this much needed improvement with Haris, and I'm very much
> > looking forward to seeing this land in the project.
> >
> > Here is his committer eligibility document for perusal:
> > https://docs.google.com/document/d/1mlO1yyLCoCSd85XeDKIxTYyboK_
> > uiOJ4Uwr6ruKTlFM/edit
> >
> > Thanks!
> > Ben
> >
> > [1] http://mesos.apache.org/documentation/latest/gpu-support/
> > [2]
> > https://docs.google.com/document/d/1nAVr0sSSpbDLrgUlAEB5hKzCl482N
> > SVk8V0D56sFMzU
> > [3]
> > https://docs.google.com/document/d/1r6Iv4Efu8v8IBrcUTjgYkvZ32WVsc
> > gYqrD07OyIglsA/
> >
>



-- 
Avinash Sridharan, Mesosphere
+1 (323) 702 5245

Re: [VOTE] Release Apache Mesos 1.2.0 (rc2)

[Neil Conway]

The perf core dump might be addressed if we backport this change:

https://reviews.apache.org/r/56611/

Although my guess is that this isn't a severe problem: for some
as-yet-unknown reason, running `perf` on the host segfaulted, which
causes the test to fail.

Neil

On Wed, Mar 1, 2017 at 11:09 AM, Vinod Kone  wrote:
> Tested on ASF CI.
>
> Saw 2 configurations fail. One was the perf core dump issue
> . Other is a known (since
> 0..28.0) flaky test with Docker fetcher plugin
> .
>
> Withholding the vote until we know the severity of the perf core dump.
>
>
> *Revision*: b9d8202a7444d0d1e49476bfc9817eb4583beaff
>
>- refs/tags/1.1.1-rc2
>
> Configuration Matrix gcc clang
> centos:7 --verbose --enable-libevent --enable-ssl autotools
> [image: Success]
> 
> [image: Not run]
> cmake
> [image: Success]
> 
> [image: Not run]
> --verbose autotools
> [image: Success]
> 
> [image: Not run]
> cmake
> [image: Success]
> 
> [image: Not run]
> ubuntu:14.04 --verbose --enable-libevent --enable-ssl autotools
> [image: Success]
> 
> [image: Failed]
> 
> cmake
> [image: Success]
> 
> [image: Success]
> 
> --verbose autotools
> [image: Success]
> 
> [image: Failed]
> 
> cmake
> [image: Success]
> 
> [image: Success]
> 
>
> On Wed, Mar 1, 2017 at 9:24 AM, Greg Mann  wrote:
>
>> I wanted to give a heads up on a flaky test failure I've encountered while
>> testing this RC: 'DockerRuntimeIsolatorTest.ROO
>> T_INTERNET_CURL_DockerDefaultEntryptRegistryPuller'. One issue related to
>> this test was resolved recently (https://issues.apache.org/
>> jira/browse/MESOS-6001), but this seems to be a separate issue (
>> https://issues.apache.org/jira/browse/MESOS-7185). I haven't had time to
>> triage yet so I'm not sure if this represents a 

Re: [VOTE] Release Apache Mesos 1.2.0 (rc2)

[Vinod Kone]

Tested on ASF CI.

Saw 2 configurations fail. One was the perf core dump issue
. Other is a known (since
0..28.0) flaky test with Docker fetcher plugin
.

Withholding the vote until we know the severity of the perf core dump.


*Revision*: b9d8202a7444d0d1e49476bfc9817eb4583beaff

   - refs/tags/1.1.1-rc2

Configuration Matrix gcc clang
centos:7 --verbose --enable-libevent --enable-ssl autotools
[image: Success]

[image: Not run]
cmake
[image: Success]

[image: Not run]
--verbose autotools
[image: Success]

[image: Not run]
cmake
[image: Success]

[image: Not run]
ubuntu:14.04 --verbose --enable-libevent --enable-ssl autotools
[image: Success]

[image: Failed]

cmake
[image: Success]

[image: Success]

--verbose autotools
[image: Success]

[image: Failed]

cmake
[image: Success]

[image: Success]


On Wed, Mar 1, 2017 at 9:24 AM, Greg Mann  wrote:

> I wanted to give a heads up on a flaky test failure I've encountered while
> testing this RC: 'DockerRuntimeIsolatorTest.ROO
> T_INTERNET_CURL_DockerDefaultEntryptRegistryPuller'. One issue related to
> this test was resolved recently (https://issues.apache.org/
> jira/browse/MESOS-6001), but this seems to be a separate issue (
> https://issues.apache.org/jira/browse/MESOS-7185). I haven't had time to
> triage yet so I'm not sure if this represents a legitimate bug, but I
> thought I'd email here to increase visibility while the vote is out.
>
> Cheers,
> Greg
>
>
> On Fri, Feb 24, 2017 at 1:14 AM, Adam Bordelon  wrote:
>
> > Dear Mesos developers and users,
> >
> > Please vote on releasing the following candidate as Apache Mesos 1.2.0.
> >
> > 1.2.0 includes the following:
> > 
> > 
> >   * 

Re: [VOTE] Release Apache Mesos 1.1.1 (rc2)

[Vinod Kone]

Tested on ASF CI.

Saw 2 configurations fail with
https://issues.apache.org/jira/browse/MESOS-7160

I think @jpeach and @bbannier were looking into this. Not sure about the
severity of the issue, so withholding my vote.


*Revision*: b9d8202a7444d0d1e49476bfc9817eb4583beaff

   - refs/tags/1.1.1-rc2

Configuration Matrix gcc clang
centos:7 --verbose --enable-libevent --enable-ssl autotools
[image: Success]

[image: Not run]
cmake
[image: Success]

[image: Not run]
--verbose autotools
[image: Success]

[image: Not run]
cmake
[image: Success]

[image: Not run]
ubuntu:14.04 --verbose --enable-libevent --enable-ssl autotools
[image: Success]

[image: Failed]

cmake
[image: Success]

[image: Success]

--verbose autotools
[image: Success]

[image: Failed]

cmake
[image: Success]

[image: Success]


On Mon, Feb 27, 2017 at 5:54 AM, Alex Rukletsov  wrote:

> Hi all,
>
> Please vote on releasing the following candidate as Apache Mesos 1.1.1.
>
> 1.1.1 includes the following:
> 
> 
> ** Bug
>   * [MESOS-6002] - The whiteout file cannot be removed correctly using
> aufs backend.
>   * [MESOS-6010] - Docker registry puller shows decode error "No response
> decoded".
>   * [MESOS-6142] - Frameworks may RESERVE for an arbitrary role.
>   * [MESOS-6360] - The handling of whiteout files in provisioner is not
> correct.
>   * [MESOS-6411] - Add documentation for CNI port-mapper plugin.
>   * [MESOS-6526] - `mesos-containerizer launch --environment` exposes
> executor env vars in `ps`.
>   * [MESOS-6571] - Add "--task" flag to mesos-execute.
>   * [MESOS-6597] - Include v1 Operator API protos in generated JAR and
> python packages.
>   * [MESOS-6606] - Reject optimized builds with libcxx before 3.9.
>   * [MESOS-6621] - SSL downgrade path will CHECK-fail when using both
> 

Re: [VOTE] Release Apache Mesos 1.2.0 (rc2)

[Greg Mann]

I wanted to give a heads up on a flaky test failure I've encountered while
testing this RC: 'DockerRuntimeIsolatorTest.ROO
T_INTERNET_CURL_DockerDefaultEntryptRegistryPuller'. One issue related to
this test was resolved recently (https://issues.apache.org/
jira/browse/MESOS-6001), but this seems to be a separate issue (
https://issues.apache.org/jira/browse/MESOS-7185). I haven't had time to
triage yet so I'm not sure if this represents a legitimate bug, but I
thought I'd email here to increase visibility while the vote is out.

Cheers,
Greg


On Fri, Feb 24, 2017 at 1:14 AM, Adam Bordelon  wrote:

> Dear Mesos developers and users,
>
> Please vote on releasing the following candidate as Apache Mesos 1.2.0.
>
> 1.2.0 includes the following:
> 
> 
>   * [MESOS-5931] - **Experimental** Support auto backend in Mesos
> Containerizer,
> prefering overlayfs then aufs. Please note that the bind backend needs
> to be
> specified explicitly through the agent flag
> '--image_provisioner_backend'
> since it requires the sandbox already existed.
>
>   * [MESOS-6402] - **Experimental** Add rlimit support to Mesos
> containerizer.
> The isolator adds support for setting POSIX resource limits (rlimits)
> for
> containers launched using the Mesos containerizer. POSIX rlimits can be
> used
> to control the resources a process can consume. See `docs/
> posix_rlimits.md`
> for details.
>
>   * [MESOS-6419] - **Experimental** Teardown unregistered frameworks. The
> master
> now treats recovered frameworks very similarly to frameworks that are
> registered
> but currently disconnected. For example, recovered frameworks will be
> reported
> via the normal "frameworks" key when querying HTTP endpoints. This
> means there
> is no longer a concept of "orphan tasks": if the master knows about a
> task, the
> task will be running under a framework. Similarly, "teardown"
> operations on
> recovered frameworks will now work correctly.
>
>   * [MESOS-6460] - **Experimental** Container Attach and Exec. This feature
> adds
> new Agent APIs for attaching a remote client to the stdin, stdout, and
> stderr
> of a running Mesos task, as well as an API for launching new processes
> inside
> the same container as a running Mesos task and attaching to its stdin,
> stdout,
> and stderr. At a high level, these APIs mimic functionality similar to
> docker
> attach and docker exec. The primary motivation for such functionality
> is to
> enable users to debug their running Mesos tasks.
>
>   * [MESOS-6758] - **Experimental** Support 'Basic' auth docker private
> registry
> on Mesos Containerizer. Until now, the mesos containerizer always
> assumed
> Bearer auth, but we now also support basic auth for private registries.
> Please
> note that the AWS ECS uses Basic authorization but it does not work yet
> due to
> the redirect issue MESOS-5172.
>
> The CHANGELOG for the release is available at:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_p
> lain;f=CHANGELOG;hb=1.2.0-rc2
> 
> 
>
> The candidate for Mesos 1.2.0 release is available at:
> https://dist.apache.org/repos/dist/dev/mesos/1.2.0-rc2/mesos-1.2.0.tar.gz
>
> The tag to be voted on is 1.2.0-rc2:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=1.2.0-rc2
>
> The MD5 checksum of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/1.2.0-rc2/mesos
> -1.2.0.tar.gz.md5
>
> The signature of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/1.2.0-rc2/mesos
> -1.2.0.tar.gz.asc
>
> The PGP key used to sign the release is here:
> https://dist.apache.org/repos/dist/release/mesos/KEYS
>
> The JAR is up in Maven in a staging repository here:
> https://repository.apache.org/content/repositories/orgapachemesos-1180
>
> Please vote on releasing this package as Apache Mesos 1.2.0!
>
> The vote is open until Wed Mar 1 18:00 PST 2017 and passes if a majority of
> at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Mesos 1.2.0
> [ ] -1 Do not release this package because ...
>
> Thanks,
> -Adam-
>

Re: Auto scaling spark driver on Mesos!

[Ashish Mehta]

Mesos might have introduced multi tenancy in new versions, but we wanted to
provide multi-tenancy in versions prior to that, via our "compute layer",
and we wanted to implement some isolation so that few machines are always
reserved for a tenant.

Apologies that I didn't explain "tagging/un-tagging", by this I mean, we
assign attributes to machines/resource in Mesos, so that they are offered
with those attributes to application running on mesos. This helps us make
use of "spark.mesos.constraints" configuration of spark, by which my spark
application only accepts offers from machine whose attributes (tag) matches
the constraints passed in.

In case of static allocation of resource, we just tag (assign attributes)
to the the resource = spark.core.max, and hence all these machines gets
used up by application. But in case of automatic scaling, my "compute
layer" should be able to know the resource requirement, to make automatic
tagging (attributes assignment)/un-tagging (removing attributes).

So reiterating my question. What is the best way to get the feedback to my
"compute layer", about spark application's requirement for auto-scalling?
How about the following

   1. Application exposing some API, or dumping event to inform the
   framework when it needs more resources
   2. Or our "compute layer" polls the Mesos API to know the resources
   consumed by the application and deduce whether auto scaling is required or
   not.

BTW Thanks for taking this talk forward.

- Ashish

On Wed, Mar 1, 2017 at 12:33 AM, Michael Gummelt 
wrote:

> I'm sorry, I still don't understand.  What functionality does this
> "compute layer" provide?  You say it provides Multi tenancy, but Mesos
> itself does that.  You also say it "keeps track of resources", but again,
> Mesos does that.  What does tagging/un-tagging resources provide?
>
>
> On Tue, Feb 28, 2017 at 12:46 AM, Ashish Mehta 
> wrote:
>
>> Yes Michael, I have tried dynamic allocation with my own Mesos cluser, it
>> works as expected and as documented!
>> But now I want to move ahead and integrate with our own "compute layer".
>> Our "compute layer"
>>
>>- provides Multi tenancy with chronos and marathon over Mesos.
>>- Manages/book-keeps all the resources on behalf of individual
>>tenant, having some quota assigned to it. It keeps track of resource by
>>tagging/un-tagging them in Mesos.
>>
>>
>> The problem with, out of the box "dynamic allocation" is that, our
>> "compute layer" doesn't know the resource utilisation of the application,
>> and can't tag/un-tag the machine automatically. If we tag all the machine
>> before running the application based on spark.cores.max, then we will not
>> be able to make use of "dynamic allocation" because the tagged machines are
>> reserved and can't be used for other applications.
>>
>> *So I want to articulate my initial query here and ask:*
>> What is the best way to get the feedback to my "compute layer", about
>> spark application's requirement for auto-scalling? How about the following
>>
>>1. Application exposing some API, or dumping event to inform the
>>framework when it needs more resources
>>2. Or our "compute layer" polls the Mesos API to know the resources
>>consumed by the application and deduce whether auto scaling is required or
>>not.
>>
>> Thanks,
>> Ashish
>>
>> On Tue, Feb 28, 2017 at 2:48 AM, Michael Gummelt 
>> wrote:
>>
>>> I assume you've looked into dynamic allocation.  What do you need that
>>> isn't provided by dynamic allocation?
>>>
>>> On Mon, Feb 27, 2017 at 4:11 AM, David J. Palaitis <
>>> david.j.palai...@gmail.com> wrote:
>>>
 by using a combination of Spark's dynamic allocation,
 http://spark.apache.org/docs/latest/job-scheduli
 ng.html#configuration-and-setup, and a framework scheduler like Cook,
 https://github.com/twosigma/Cook/tree/master/spark, you can achieve
 the desired auto-scaling effect without the overhead of managing
 roles/constraints in mesos.  i'd be happy to discuss this in more detail if
 you decide to give it a try.

 On Mon, Feb 27, 2017 at 3:14 AM, Ashish Mehta  wrote:

> Hi,
>
> We want to move to auto-scaling of spark driver, where in more
> resources are added into the available resources for "spark driver" based
> on requirement. The requirement can increase/decrease based on multiple 
> SQL
> queries being done over REST server, or number of queries with multiple
> user over thrift server over Spark (HiveServer2).
>
> *Existing approach with static number of resources:*
> We have a very large pool of resources, but my "spark driver" is
> allocated limited amount of "static" resource, and we achieve this by
> following
>
>1. While running the application I tag machine in Mesos with the
>name of my application, so that the