Re: Communicate with a container while using Mesos unified container runtime

[Karan Pradhan]

Thanks for all the ideas, I'll try dcos cli after upgrading mesos.

On Fri, Mar 23, 2018 at 2:35 PM, Daemeon Reiydelle <daeme...@gmail.com>
wrote:

> For what I understand to be your use case, I will have queuing services
> that the container queries for its next task. E.g. Kafka "queues"
>
>
>
> Please pardon typo's  ... sent from mobile
>
> daeme...@gmail.com
> USA MOBILE: 415.501.0198 <(415)%20501-0198> (California)
>
>  Original message 
> From: Gilbert Song <gilb...@mesosphere.io>
> Date: 3/23/18 13:47 (GMT-08:00)
> To: user <user@mesos.apache.org>
> Subject: Re: Communicate with a container while using Mesos unified
> container runtime
>
> No, this feature based on Container Attach/Exec
> <https://issues.apache.org/jira/browse/MESOS-6460>, which was included
> starting from Mesos 1.2.0. I would recommend an upgrade to Mesos 1.4.1 or
> 1.3.2.
>
> On Thu, Mar 22, 2018 at 5:46 PM, Karan Pradhan <karanprad...@gmail.com>
> wrote:
>
>>
>>
>> On 2018/03/22 23:16:06, Gilbert Song <gilb...@apache.org> wrote:
>> > Hi Karan,
>> >
>> > It does not seem to me that launching more mesos containers would add
>> more
>> > overheads.
>> >
>> > If you want to achieve *docker exec* for debugging purpose, Mesos
>> supports
>> > that (not in Mesos CLI yet /cc Armand and Kevin), but you could still
>> rely
>> > on dc/os CLI
>> > <https://docs.mesosphere.com/1.11/monitoring/debugging/task-exec/> to
>> do
>> > that given you have the taskId.
>> >
>> > Gilbert
>> >
>> > On Wed, Mar 21, 2018 at 12:08 PM, Karan Pradhan <karanprad...@gmail.com
>> >
>> > wrote:
>> >
>> > >
>> > >
>> > > On 2018/03/21 18:06:48, Gilbert Song <gilb...@apache.org> wrote:
>> > > > Hi Karan,
>> > > >
>> > > > Before figuring out some ways to achieve this with Mesos, I would
>> like to
>> > > > better understand your use cases.
>> > > >
>> > > > Do you mean you rely on `docker attach/exec` to send commands to an
>> > > > existing running container?
>> > > >
>> > > > Is there any reason that keeps you from launching a container for
>> each
>> > > > batch job?
>> > > >
>> > > > Gilbert
>> > > >
>> > > > On Wed, Mar 21, 2018 at 10:29 AM, karanprad...@gmail.com <
>> > > > karanprad...@gmail.com> wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > I was docker for running my batch job in which I would follow this
>> > > > > approach:
>> > > > >
>> > > > > 1. Start the docker container
>> > > > > 2. Send commands to the running Docker container with the help of
>> > > docker
>> > > > > python client for each batch of objects.
>> > > > > 3. After all the batches are processed by the docker, shut down
>> the
>> > > > > container.
>> > > > >
>> > > > > I wanted to achieve the same with the help of Mesos and Marathon
>> to
>> > > spin
>> > > > > up containers and submit commands per batch.
>> > > > > But looking a the documents it looks like that this behavior is
>> not
>> > > > > achievable as when Mesos spin up a Docker container with the help
>> of
>> > > Mesos
>> > > > > containerizer and docker/runtime isolation you can submit only one
>> > > command
>> > > > > after which the Sesos framework is killed.
>> > > > >
>> > > > > It would be great if someone could point me to a way to achieve
>> this
>> > > using
>> > > > > Mesos containerizer?
>> > > > >
>> > > > > Thanks,
>> > > > > Karan
>> > > > >
>> > > >
>> > > Hi Gilbert,
>> > > Thanks for taking time answering my question.
>> > >
>> > > Yes as you mentioned I use docker exec to run commands in the
>> container.
>> > > There is no particular reason why we don't run new docker. Would that
>> add
>> > > overhead if I had multiple batches which need to be processed?
>> > >
>> > > Do you know if docker exec is possible on a mesos container running
>> with
>> > > docker/runtime isolation?
>> > >
>> > > Thanks,
>> > > Karan
>> > >
>> >
>> Thanks Gilbert.
>> I have built mesos 1.1.1 from the apache distribution, would dc/os cli
>> work for this version too?
>>
>
>

Mesos scalability

[Karan Pradhan]

Hi All,

I had the following questions:
1.
I was wondering if it is possible to have multiple Mesos masters as elected 
masters in a Mesos cluster so that the load can be balanced amongst the 
masters. Is there a way to achieve this?
In general, can there be a load balancer for the Mesos masters?

2.
I have seen spikes in the Mesos event queues while running spark SQL workloads 
with multiple stages. So I was wondering what is a better way to handle these 
scalability issues. I noticed that compute intensive machines were able to deal 
with those workloads better. Is there a particular hardware requirement or 
requirement for the number of masters for scaling a Mesos cluster horizontally? 
After reading success stories which mention that Mesos is deployed for ~10K 
machines, I was curious about the hardware used and the number of masters in 
this case. 

It would be awesome if I could get some insight into these questions.

Thanks,
Karan

Re: Communicate with a container while using Mesos unified container runtime

[Karan Pradhan]

On 2018/03/21 17:51:58, Harold Dost  wrote: 
> What gain are you looking to get from having the singular container. If you
> desire a shared context you could consider listening on a port and sending
> them over http. Otherwise I would say just launch each of your tasks
> separately.
> 
> 
> 
> Harold Dost | @hdost
> 
> 
> 
> On Wed, Mar 21, 2018, 18:29 karanprad...@gmail.com 
> wrote:
> 
> > Hi,
> >
> > I was docker for running my batch job in which I would follow this
> > approach:
> >
> > 1. Start the docker container
> > 2. Send commands to the running Docker container with the help of docker
> > python client for each batch of objects.
> > 3. After all the batches are processed by the docker, shut down the
> > container.
> >
> > I wanted to achieve the same with the help of Mesos and Marathon to spin
> > up containers and submit commands per batch.
> > But looking a the documents it looks like that this behavior is not
> > achievable as when Mesos spin up a Docker container with the help of Mesos
> > containerizer and docker/runtime isolation you can submit only one command
> > after which the Sesos framework is killed.
> >
> > It would be great if someone could point me to a way to achieve this using
> > Mesos containerizer?
> >
> > Thanks,
> > Karan
> >
> 
Hi Harold,
Thanks for taking time answering my question.


There is no particular reason why we don't run new docker. Would that add 
overhead if I had multiple batches which need to be processed?
Also, I have a constraint that the docker must run on a particular host for all 
the batches I have. 

Thanks,
Karan

Re: Communicate with a container while using Mesos unified container runtime

[Karan Pradhan]

On 2018/03/21 18:06:48, Gilbert Song  wrote: 
> Hi Karan,
> 
> Before figuring out some ways to achieve this with Mesos, I would like to
> better understand your use cases.
> 
> Do you mean you rely on `docker attach/exec` to send commands to an
> existing running container?
> 
> Is there any reason that keeps you from launching a container for each
> batch job?
> 
> Gilbert
> 
> On Wed, Mar 21, 2018 at 10:29 AM, karanprad...@gmail.com <
> karanprad...@gmail.com> wrote:
> 
> > Hi,
> >
> > I was docker for running my batch job in which I would follow this
> > approach:
> >
> > 1. Start the docker container
> > 2. Send commands to the running Docker container with the help of docker
> > python client for each batch of objects.
> > 3. After all the batches are processed by the docker, shut down the
> > container.
> >
> > I wanted to achieve the same with the help of Mesos and Marathon to spin
> > up containers and submit commands per batch.
> > But looking a the documents it looks like that this behavior is not
> > achievable as when Mesos spin up a Docker container with the help of Mesos
> > containerizer and docker/runtime isolation you can submit only one command
> > after which the Sesos framework is killed.
> >
> > It would be great if someone could point me to a way to achieve this using
> > Mesos containerizer?
> >
> > Thanks,
> > Karan
> >
> 
Hi Gilbert,
Thanks for taking time answering my question.

Yes as you mentioned I use docker exec to run commands in the container.
There is no particular reason why we don't run new docker. Would that add 
overhead if I had multiple batches which need to be processed?

Do you know if docker exec is possible on a mesos container running with 
docker/runtime isolation?

Thanks,
Karan