Re: Mesos Infrastructure - Intern

2018-11-20 Thread Thodoris Zois
Hello Stephan,

Thank you very much!


> On 15 Nov 2018, at 16:13, Kraft, Stephan  wrote:
> 
> Hi Thodoris,
> 
> Feel free to apply for one of our open positions if this is of interest to 
> you:
> 
> (Junior) Cloud Engineer for SAP HANA Job
> https://jobs.sap.com/job/WalldorfSt_-Leon-Rot-%28Junior%29-Cloud-Engineer-for-SAP-HANA-Job-BW/499669101/?locale=en_US
> 
> 
> 
> Cheers,
> Stephan
> 
> 
> On 11/14/18, 11:56 PM, "Thodoris Zois"  wrote:
> 
>Hello list,
> 
>I am sorry about the irrelevant content, but I think this is the best 
> place to ask! 
> 
>Currently I am a postgraduate student at Vrije University of Amsterdam and 
> I am looking for a summer internship. In my BSc. thesis I got a lot of 
> hands-on experience on Mesos among other interesting stuff. Since many people 
> here come from companies that use Mesos and I would really like to continue 
> working on that direction, if there is any open position in Europe, please 
> let me know so I can apply!
> 
>Thank you very much,
>Thodoris Zois
> 



Mesos Infrastructure - Intern

2018-11-14 Thread Thodoris Zois
Hello list,

I am sorry about the irrelevant content, but I think this is the best place to 
ask! 

Currently I am a postgraduate student at Vrije University of Amsterdam and I am 
looking for a summer internship. In my BSc. thesis I got a lot of hands-on 
experience on Mesos among other interesting stuff. Since many people here come 
from companies that use Mesos and I would really like to continue working on 
that direction, if there is any open position in Europe, please let me know so 
I can apply!

Thank you very much,
Thodoris Zois

Re: Resource offers - DRF - Mesos

2018-05-22 Thread Thodoris Zois
Thank you very much,

This is exactly the answer I was searching for!

- Thodoris

> On 22 May 2018, at 21:10, Joseph Wu <jos...@mesosphere.io> wrote:
> 
> 1) DRF is based on the _current_ allocation of resources (from the master's 
> perspective) rather than a historical allocation of resources.
> 
> 2) So when a new cluster is started, all frameworks will have a current 
> allocation of 0.  And assuming all else (like quotas, roles, and weights) are 
> equivalent (or not set to anything), then your 2 frameworks would receive 
> roughly equal shares of offers.
> 
> 3) As of right now, there is no way for the framework to directly influence 
> the number of offers received in a single call.  The best approach to getting 
> offers on multiple machines is to hold onto the offers (i.e. not accepting 
> nor declining them) until your necessary conditions have been met.
> 
>> On Tue, May 22, 2018 at 2:56 AM, Thodoris Zois <z...@ics.forth.gr> wrote:
>> Hello list,
>> 
>> I have some questions about resource offers for Mesos and I am
>> experiencing some problems that I hope somebody will be able to help.
>> 
>> 1) The allocation module of Mesos master uses DRF (according to
>> previous allocation history) and decides which framework will get an
>> offer, and how many resources will be offered. Is this right?
>> 
>> 2) Assume that a Mesos cluster starts for the very first time and 2
>> frameworks join. None of the frameworks has a job to submit, they just
>> wait and get offers. What is the policy to send resource offers since
>> master does not know anything about previous allocations?
>> 
>> 3) I got a Mesos cluster with 5 machines and 1 framework only. Is there
>> any way to force Mesos send everytime all the five resource offers to
>> my framework? I have seen that when the framework registers for the
>> first time, it gets a list of offers that include all 5 machines.
>> However, if it does not accept them the next round of offers contains
>> only 2 or even 1 machine, depending on the time that framework has
>> declined.
>> 
>> Thank you very much for your response,
>> Any help is appreciated!
>> 
>> - Thodoris
> 


Resource offers - DRF - Mesos

2018-05-22 Thread Thodoris Zois
Hello list,

I have some questions about resource offers for Mesos and I am
experiencing some problems that I hope somebody will be able to help.

1) The allocation module of Mesos master uses DRF (according to
previous allocation history) and decides which framework will get an
offer, and how many resources will be offered. Is this right?

2) Assume that a Mesos cluster starts for the very first time and 2
frameworks join. None of the frameworks has a job to submit, they just
wait and get offers. What is the policy to send resource offers since
master does not know anything about previous allocations?

3) I got a Mesos cluster with 5 machines and 1 framework only. Is there
any way to force Mesos send everytime all the five resource offers to
my framework? I have seen that when the framework registers for the
first time, it gets a list of offers that include all 5 machines.
However, if it does not accept them the next round of offers contains
only 2 or even 1 machine, depending on the time that framework has
declined.

Thank you very much for your response,
Any help is appreciated!

- Thodoris


MesosCon 2018 - Any Plans?

2018-02-28 Thread Thodoris Zois
Hello list,

Are there any plans for a MesosCon in 2018?

Thank you very much,
Thodoris Zois


Re: Mesos , Java & hardware isolation

2017-11-07 Thread Thodoris Zois
Hello,

I don't really know if i can help you but i used Docker containers in Mesos 
with isolation and everything was perfect. So are you using Mesos 
containerizer? Did you enable cgroups/mem, cgroups/cpu on each agent? As it 
concerns CPU, cgroups do not limit the anount of CPU if its the only running 
application. In order to limit exactly CPU you need to enable cfs on each 
agent. 

Thodoris

> On 7 Nov 2017, at 15:28, deadbrain  wrote:
> 
> Hi all,
> 
> I expect to be at the right place for my question...
> 
> We are hosting Java micro services exposed as docker containers (Spring Boot 
> applications inside) inside Mesos
> 
> We face 2 major problems:
> 
> - hardware isolation (number of cores seen by Java Virtual Machines) makes 
> our applications go crazy because JVM sees all cores and not the only ones 
> given in the Mesos config parameters as a consequence GC heuristics are wrong 
> and  some config values in some libraries are false too (Undertow web server 
> spawns 8*number of cores as workers thread pool)...
> 
> In Linux since the 2.4.19 kernel we have the per process namespace option to 
> limit the hardware vision of any container... What would  be the solution on 
> Mesos ?
> 
> 
> - CPU starvation : some micro services do not seem to have the required CPU 
> time to work, what kind of advices could you give me to have a better 
> understanding regarding this problem
> 
> Statement: we can see very long partial GC times (about 4 seconds) , standard 
> times would be about 50ms
> 
> 
> Thanks for your help
> 
> Kind regards
> 
> Jerome
> 



Cache Isolation - Postgres

2017-10-03 Thread Thodoris Zois
Hello,

I have a problem with a task of postgres i am running. The task reads 6.5GBs of 
data and if it has 7GB of memory it performs in about 20-30 seconds, else it 
takes more than 40-50 seconds to finish. This was not measured on Mesos btw. 

The problem is that on Mesos even if i am assigning to task 1GB of memory, or 
30GBs the execution time remains the same. I have enabled cgroups/mem In the 
same container i perform two runs. Before the first run i drop_caches and the 
execution time no matter the amount of memory is 40 seconds. After i perform 
the second run. The execution time no matter the amount of memory remains the 
same at 20-30 seconds i used `EXPLAIN` on the query, and no matter the 
resources given the hits are the same. On the first run it makes about 100 for 
example and on the second about 1000.

I can't determine what the problem is, and i have to figure it out. Maybe there 
is no isolation for buffer I/O cache? I hope that someone more experienced than 
me can notice the problem.

Its like it caches all the data. I have also tried assigning 64MBs of memory. 
The hits in cache remains the same, and so the execution time on both runs. 
Maybe i should enable another isolator of cgroups?

Thank you,
Zois Theodoros
--
Institute of Computer Science
FORTH-ICS Hellas


Re: Ports and Tasks

2017-09-20 Thread Thodoris Zois
Seems that it is :) 

Thank you!


> On 20 Sep 2017, at 17:00, tommy xiao <xia...@gmail.com> wrote:
> 
> ENV is good fit.
> 
> 2017-09-20 15:53 GMT+08:00 Thodoris Zois <z...@ics.forth.gr 
> <mailto:z...@ics.forth.gr>>:
> Hello,
> 
> Thank you for the provided links, i am using my own framework..
> 
> Thodoris
> 
> On 20 Sep 2017, at 03:15, Sunil Shah <suni...@yelp.com 
> <mailto:suni...@yelp.com>> wrote:
> 
>> Hi Thodoris,
>> 
>> What framework are you using? Marathon makes this port available as an 
>> environment variable, you may want to do similar: docs 
>> <https://mesosphere.github.io/marathon/docs/task-environment-vars.html>, 
>> code 
>> <https://github.com/mesosphere/marathon/blob/master/src/main/scala/mesosphere/mesos/EnvironmentHelper.scala#L25>.
>> 
>> Sunil
>> 
>> On Tue, Sep 19, 2017 at 4:46 PM, Thodoris Zois <z...@ics.forth.gr 
>> <mailto:z...@ics.forth.gr>> wrote:
>> Hello list,
>> 
>> I have a weird problem, and i am wondering if there is any easy way to solve 
>> it.
>> 
>> I am running some tasks (docker images using mesos contaunerizer) that each 
>> one starts a mysql server, does some queries and closes the server.
>> 
>> Every server is listening to the same port. And that is the problem. In 
>> Mesos offer i allocate a port for each task, but how i can use it in the 
>> script that docker file (CMD script.sh) executes when starts?
>> 
>> Any help would be appreciated!
>> 
>> Thank you very much,
>> Thodoris
>> 
> 
> 
> 
> -- 
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com <http://gmail.com/>


Re: Ports and Tasks

2017-09-20 Thread Thodoris Zois
Hello,

Thank you for the provided links, i am using my own framework..

Thodoris

> On 20 Sep 2017, at 03:15, Sunil Shah <suni...@yelp.com> wrote:
> 
> Hi Thodoris,
> 
> What framework are you using? Marathon makes this port available as an 
> environment variable, you may want to do similar: docs, code.
> 
> Sunil
> 
>> On Tue, Sep 19, 2017 at 4:46 PM, Thodoris Zois <z...@ics.forth.gr> wrote:
>> Hello list,
>> 
>> I have a weird problem, and i am wondering if there is any easy way to solve 
>> it.
>> 
>> I am running some tasks (docker images using mesos contaunerizer) that each 
>> one starts a mysql server, does some queries and closes the server.
>> 
>> Every server is listening to the same port. And that is the problem. In 
>> Mesos offer i allocate a port for each task, but how i can use it in the 
>> script that docker file (CMD script.sh) executes when starts?
>> 
>> Any help would be appreciated!
>> 
>> Thank you very much,
>> Thodoris
> 


Ports and Tasks

2017-09-19 Thread Thodoris Zois
Hello list,

I have a weird problem, and i am wondering if there is any easy way to solve it.

I am running some tasks (docker images using mesos contaunerizer) that each one 
starts a mysql server, does some queries and closes the server. 

Every server is listening to the same port. And that is the problem. In Mesos 
offer i allocate a port for each task, but how i can use it in the script that 
docker file (CMD script.sh) executes when starts?

Any help would be appreciated!

Thank you very much,
Thodoris 


Re: Containerizers & Isolation

2017-09-14 Thread Thodoris Zois
Thank you very much for your response!

When talking about executors i am referring only to default executor and 
command executor and not the custom one.
You say that there should be only one executor per container. By that you mean 
that you can run multiple executors under the same container but there won’t be 
any isolation between those executors or that it MUST be one executor per 
container? 

When using command executor always different executors go under different 
containers, right?

When using the default executor?

Thank you,
Thodoris


> On 14 Sep 2017, at 18:53, Jie Yu <yujie@gmail.com> wrote:
> 
> - When someone is using the Docker Containerizer what kind of isolation can 
> be used and how Mesos handles it?
> 
> It's the default isolation that Docker engine provides. You can alter some of 
> those using ContainerInfo.DockerInfo.
> 
> - Custom isolators can be used both for Docker Containerizer and Mesos 
> Containerizer or only in Mesos Containerizer?
> 
> Custom isolators only work for Mesos Containerizer currently.
> 
> - Is there any isolation between the executors under the same container when 
> using Mesos Containerizer?
> 
> There should be only one executor per container.
> 
>  - A task is launched by command executor, default mesos executor, or custom 
> executor. In each case task goes under the container of the executor? or a 
> task and an executor are two different containers?
> 
> Tasks go under the container of the executor.
> 
> - Jie 
>  
> 
> On Thu, Sep 14, 2017 at 8:18 AM, Thodoris Zois <z...@ics.forth.gr 
> <mailto:z...@ics.forth.gr>> wrote:
> Hello list,
> 
> I have 4 questions, and i would be glad if someone can help..
> 
> - When someone is using the Docker Containerizer what kind of isolation can 
> be used and how Mesos handles it?
> 
> - Custom isolators can be used both for Docker Containerizer and Mesos 
> Containerizer or only in Mesos Containerizer?
> 
> - Is there any isolation between the executors under the same container when 
> using Mesos Containerizer?
> 
> - A task is launched by command executor, default mesos executor, or custom 
> executor. In each case task goes under the container of the executor? or a 
> task and an executor are two different containers?
>   Because i saw an image (attached) on the book Apache Mesos Essentials and 
> it actually confused me..
> 
> Thank you very much,
> Thodoris
> 
> 
> 



Containerizers & Isolation

2017-09-14 Thread Thodoris Zois
Hello list,

I have 4 questions, and i would be glad if someone can help..

- When someone is using the Docker Containerizer what kind of isolation can be 
used and how Mesos handles it?

- Custom isolators can be used both for Docker Containerizer and Mesos 
Containerizer or only in Mesos Containerizer?

- Is there any isolation between the executors under the same container when 
using Mesos Containerizer?

- A task is launched by command executor, default mesos executor, or custom 
executor. In each case task goes under the container of the executor? or a task 
and an executor are two different containers?
  Because i saw an image (attached) on the book Apache Mesos Essentials and it 
actually confused me..

Thank you very much,
Thodoris



Re: Docker run command - Mesos Containerizer

2017-09-13 Thread Thodoris Zois
I will try it! Thank you…


> On 13 Sep 2017, at 10:32, j...@is-land.com.tw wrote:
> 
> 
> 
> On 2017-09-13 15:17, Thodoris Zois <z...@ics.forth.gr 
> <mailto:z...@ics.forth.gr>> wrote: 
>> Yes, i have made my own framework to run docker images under Mesos 
>> Containerizer.
>> But i don’t know how to make the image privileged when running, and i need 
>> to pass a variable at the image…
>> 
>> Thodoris
>> 
>>> On 13 Sep 2017, at 09:27, j...@is-land.com.tw <mailto:j...@is-land.com.tw> 
>>> wrote:
>>> 
>>> 
>>> 
>>> On 2017-09-13 12:13, Thodoris Zois <z...@ics.forth.gr 
>>> <mailto:z...@ics.forth.gr>> wrote: 
>>>> Hello,
>>>> 
>>>> Yes you are probably right.. what i am trying to do is to make the 
>>>> container image privileged, and also i would like to some some 
>>>> environmental variables to the script running by the docker image. Any 
>>>> ideas? 
>>>> 
>>>>> On 13 Sep 2017, at 05:36, "j...@is-land.com.tw 
>>>>> <mailto:j...@is-land.com.tw>"<j...@is-land.com.tw 
>>>>> <mailto:j...@is-land.com.tw>> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>>> On 2017-09-12 19:22, Thodoris Zois <z...@ics.forth.gr 
>>>>>> <mailto:z...@ics.forth.gr>> wrote: 
>>>>>> Hello,
>>>>>> 
>>>>>> I am using, Mesos Containerizer with Docker Images. 
>>>>>> Is there any way to specify my command for docker run? I would like to 
>>>>>> pass some arguments for the docker image, like —privileged and an 
>>>>>> enviromental variable
>>>>>> 
>>>>>> Thank you,
>>>>>> Thodoris
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> I have the impression that can't use the `docker run` command.
>>>>> 
>>>>> Need to use the running framework task configuration.
>>>>> 
>>>>> For more details please refer to:
>>>>> https://mesosphere.github.io/marathon/docs/native-docker.html
>>>> 
>>>> 
>>> Hi,
>>> 
>>> I don't have any no idea.
>>> 
>>> How did you make the container image privileged?
>>> 
>>> Did you Create a custom Framework for running Docker images?
>>> 
>> 
>> 
> 
> Maybe can be traced the Marathon Framework source code



Re: Docker run command - Mesos Containerizer

2017-09-13 Thread Thodoris Zois
Yes, i have made my own framework to run docker images under Mesos 
Containerizer.
But i don’t know how to make the image privileged when running, and i need to 
pass a variable at the image…

Thodoris

> On 13 Sep 2017, at 09:27, j...@is-land.com.tw wrote:
> 
> 
> 
> On 2017-09-13 12:13, Thodoris Zois <z...@ics.forth.gr> wrote: 
>> Hello,
>> 
>> Yes you are probably right.. what i am trying to do is to make the container 
>> image privileged, and also i would like to some some environmental variables 
>> to the script running by the docker image. Any ideas? 
>> 
>>> On 13 Sep 2017, at 05:36, "j...@is-land.com.tw"<j...@is-land.com.tw> wrote:
>>> 
>>> 
>>> 
>>>> On 2017-09-12 19:22, Thodoris Zois <z...@ics.forth.gr> wrote: 
>>>> Hello,
>>>> 
>>>> I am using, Mesos Containerizer with Docker Images. 
>>>> Is there any way to specify my command for docker run? I would like to 
>>>> pass some arguments for the docker image, like —privileged and an 
>>>> enviromental variable
>>>> 
>>>> Thank you,
>>>> Thodoris
>>> 
>>> Hi,
>>> 
>>> I have the impression that can't use the `docker run` command.
>>> 
>>> Need to use the running framework task configuration.
>>> 
>>> For more details please refer to:
>>> https://mesosphere.github.io/marathon/docs/native-docker.html
>> 
>> 
> Hi,
> 
> I don't have any no idea.
> 
> How did you make the container image privileged?
> 
> Did you Create a custom Framework for running Docker images?
> 



Re: Docker run command - Mesos Containerizer

2017-09-12 Thread Thodoris Zois
Hello,

Yes you are probably right.. what i am trying to do is to make the container 
image privileged, and also i would like to some some environmental variables to 
the script running by the docker image. Any ideas? 

> On 13 Sep 2017, at 05:36, "j...@is-land.com.tw"<j...@is-land.com.tw> wrote:
> 
> 
> 
>> On 2017-09-12 19:22, Thodoris Zois <z...@ics.forth.gr> wrote: 
>> Hello,
>> 
>> I am using, Mesos Containerizer with Docker Images. 
>> Is there any way to specify my command for docker run? I would like to pass 
>> some arguments for the docker image, like —privileged and an enviromental 
>> variable
>> 
>> Thank you,
>> Thodoris
> 
> Hi,
> 
> I have the impression that can't use the `docker run` command.
> 
> Need to use the running framework task configuration.
> 
> For more details please refer to:
> https://mesosphere.github.io/marathon/docs/native-docker.html



Docker run command - Mesos Containerizer

2017-09-12 Thread Thodoris Zois
Hello,

I am using, Mesos Containerizer with Docker Images. 
Is there any way to specify my command for docker run? I would like to pass 
some arguments for the docker image, like —privileged and an enviromental 
variable

Thank you,
Thodoris

Re: TASK_FAILED - Mesos Container Images

2017-09-07 Thread Thodoris Zois
No problem! I am glad that you tried to help me.. Thank you!

> On 7 Sep 2017, at 18:21, Olivier Sallou <olivier.sal...@irisa.fr> wrote:
> 
> 
> 
> On 09/07/2017 04:58 PM, Thodoris Zois wrote:
>> I am using the Mesos Containerizer and i run docker images. I don’t have any 
>> proto DockerInfo.. I use Protos.ContainerInfo and Protos.Image
> oh sorry, though you were using docker containerizer
> I don't know what are limitations for mesos containerizer regarding /proc 
> etc... access and how to override this.
> 
>> 
>> 
>>> On 7 Sep 2017, at 17:48, Olivier Sallou <olivier.sal...@irisa.fr 
>>> <mailto:olivier.sal...@irisa.fr>> wrote:
>>> 
>>> 
>>> On 09/07/2017 04:44 PM, Thodoris Zois wrote:
>>>> Actually my issue is that i want to set some flags on docker run so i can 
>>>> give privileges to the docker image, using —privileged. In the Dockerfile 
>>>> i am using CMD script.sh in which my command fails due to privileges.
>>> in mesos.proto there is in DockerInfo the "privileged" parameter
>>> The "parameters" parameter gives you possibility to specify arbitrary ones 
>>> but this is not advised for future compatibility (and could depend on 
>>> docker versions)
>>>> 
>>>> Thodoris 
>>>> 
>>>>> On 7 Sep 2017, at 17:40, Olivier Sallou <olivier.sal...@irisa.fr 
>>>>> <mailto:olivier.sal...@irisa.fr>> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> On 09/06/2017 06:40 PM, Thodoris Zois wrote:
>>>>>> Yes you are right, this is the problem. I aso noticed it. Docker image 
>>>>>> has no privileges so script can't execute this command. Is there any way 
>>>>>> to use arguments in proto? Or i have to set command to the task? And set 
>>>>>> shell to true? 
>>>>> personally, I set
>>>>> task.command.shell to True
>>>>> and task.command.value = "myscript to execute"
>>>>> 
>>>>> "myscript" is the bash commands to execute.
>>>>> 
>>>>> This can be used to override default entrypoint if this is your issue
>>>>> 
>>>>> Olivier
>>>>>> 
>>>>>> Thodoris
>>>>>> 
>>>>>>> On 6 Sep 2017, at 18:26, James Peach <jor...@gmail.com 
>>>>>>> <mailto:jor...@gmail.com>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>>> On Sep 6, 2017, at 4:41 AM, Thodoris Zois <z...@ics.forth.gr 
>>>>>>>> <mailto:z...@ics.forth.gr>> wrote:
>>>>>>>> 
>>>>>>>> Hello, 
>>>>>>>> 
>>>>>>>> I am using the Mesos Containerizer with Docker Images. The problem is 
>>>>>>>> that whenever a container exits my task gets TASK_FAILED because the 
>>>>>>>> container exits with ‘1’.
>>>>>>>> My docker file invokes a shell script via CMD /script.sh.
>>>>>>>> 
>>>>>>>> My protobuff can be found below:
>>>>>>>> https://pastebin.com/1agjAFdm <https://pastebin.com/1agjAFdm>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> The agent log can be found here:
>>>>>>>> https://pastebin.com/Q6qndVuU <https://pastebin.com/Q6qndVuU>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> The stderr and stdout from the UI can be found here:
>>>>>>>> stdout: https://pastebin.com/SkDDUDDJ <https://pastebin.com/SkDDUDDJ>
>>>>>>>> 
>>>>>>>> stderr: https://pastebin.com/t2TFgQ4G <https://pastebin.com/t2TFgQ4G>
>>>>>>> /execute_blast.sh: line 4: /proc/sys/vm/drop_caches: Read-only file 
>>>>>>> system
>>>>>>> 
>>>>>>> 
>>>>>>> Does your task exit when this happens?
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> -- 
>>>>> Olivier Sallou
>>>>> IRISA / University of Rennes 1
>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>>>> Tel: 02.99.84.71.95
>>>>> 
>>>>> gpg key id: 4096R/326D8438  (keyring.debian.org 
>>>>> <http://keyring.debian.org/>)
>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
>>>> 
>>> 
>>> -- 
>>> Olivier Sallou
>>> IRISA / University of Rennes 1
>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>> Tel: 02.99.84.71.95
>>> 
>>> gpg key id: 4096R/326D8438  (keyring.debian.org 
>>> <http://keyring.debian.org/>)
>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
>>> 
>> 
> 
> -- 
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95
> 
> gpg key id: 4096R/326D8438  (keyring.debian.org <http://keyring.debian.org/>)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
> 
> 



Re: TASK_FAILED - Mesos Container Images

2017-09-07 Thread Thodoris Zois
I am using the Mesos Containerizer and i run docker images. I don’t have any 
proto DockerInfo.. I use Protos.ContainerInfo and Protos.Image


> On 7 Sep 2017, at 17:48, Olivier Sallou <olivier.sal...@irisa.fr> wrote:
> 
> 
> On 09/07/2017 04:44 PM, Thodoris Zois wrote:
>> Actually my issue is that i want to set some flags on docker run so i can 
>> give privileges to the docker image, using —privileged. In the Dockerfile i 
>> am using CMD script.sh in which my command fails due to privileges.
> in mesos.proto there is in DockerInfo the "privileged" parameter
> The "parameters" parameter gives you possibility to specify arbitrary ones 
> but this is not advised for future compatibility (and could depend on docker 
> versions)
>> 
>> Thodoris 
>> 
>>> On 7 Sep 2017, at 17:40, Olivier Sallou <olivier.sal...@irisa.fr 
>>> <mailto:olivier.sal...@irisa.fr>> wrote:
>>> 
>>> 
>>> 
>>> On 09/06/2017 06:40 PM, Thodoris Zois wrote:
>>>> Yes you are right, this is the problem. I aso noticed it. Docker image has 
>>>> no privileges so script can't execute this command. Is there any way to 
>>>> use arguments in proto? Or i have to set command to the task? And set 
>>>> shell to true? 
>>> personally, I set
>>> task.command.shell to True
>>> and task.command.value = "myscript to execute"
>>> 
>>> "myscript" is the bash commands to execute.
>>> 
>>> This can be used to override default entrypoint if this is your issue
>>> 
>>> Olivier
>>>> 
>>>> Thodoris
>>>> 
>>>>> On 6 Sep 2017, at 18:26, James Peach <jor...@gmail.com 
>>>>> <mailto:jor...@gmail.com>> wrote:
>>>>> 
>>>>> 
>>>>>> On Sep 6, 2017, at 4:41 AM, Thodoris Zois <z...@ics.forth.gr 
>>>>>> <mailto:z...@ics.forth.gr>> wrote:
>>>>>> 
>>>>>> Hello, 
>>>>>> 
>>>>>> I am using the Mesos Containerizer with Docker Images. The problem is 
>>>>>> that whenever a container exits my task gets TASK_FAILED because the 
>>>>>> container exits with ‘1’.
>>>>>> My docker file invokes a shell script via CMD /script.sh.
>>>>>> 
>>>>>> My protobuff can be found below:
>>>>>> https://pastebin.com/1agjAFdm <https://pastebin.com/1agjAFdm>
>>>>>> 
>>>>>> 
>>>>>> The agent log can be found here:
>>>>>> https://pastebin.com/Q6qndVuU <https://pastebin.com/Q6qndVuU>
>>>>>> 
>>>>>> 
>>>>>> The stderr and stdout from the UI can be found here:
>>>>>> stdout: https://pastebin.com/SkDDUDDJ <https://pastebin.com/SkDDUDDJ>
>>>>>> 
>>>>>> stderr: https://pastebin.com/t2TFgQ4G <https://pastebin.com/t2TFgQ4G>
>>>>> /execute_blast.sh: line 4: /proc/sys/vm/drop_caches: Read-only file system
>>>>> 
>>>>> 
>>>>> Does your task exit when this happens?
>>>>> 
>>>>> 
>>> 
>>> -- 
>>> Olivier Sallou
>>> IRISA / University of Rennes 1
>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>> Tel: 02.99.84.71.95
>>> 
>>> gpg key id: 4096R/326D8438  (keyring.debian.org 
>>> <http://keyring.debian.org/>)
>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
>> 
> 
> -- 
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95
> 
> gpg key id: 4096R/326D8438  (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
> 



Re: TASK_FAILED - Mesos Container Images

2017-09-07 Thread Thodoris Zois
Actually my issue is that i want to set some flags on docker run so i can give 
privileges to the docker image, using —privileged. In the Dockerfile i am using 
CMD script.sh in which my command fails due to privileges.

Thodoris 

> On 7 Sep 2017, at 17:40, Olivier Sallou <olivier.sal...@irisa.fr> wrote:
> 
> 
> 
> On 09/06/2017 06:40 PM, Thodoris Zois wrote:
>> Yes you are right, this is the problem. I aso noticed it. Docker image has 
>> no privileges so script can't execute this command. Is there any way to use 
>> arguments in proto? Or i have to set command to the task? And set shell to 
>> true? 
> personally, I set
> task.command.shell to True
> and task.command.value = "myscript to execute"
> 
> "myscript" is the bash commands to execute.
> 
> This can be used to override default entrypoint if this is your issue
> 
> Olivier
>> 
>> Thodoris
>> 
>>> On 6 Sep 2017, at 18:26, James Peach <jor...@gmail.com> wrote:
>>> 
>>> 
>>>> On Sep 6, 2017, at 4:41 AM, Thodoris Zois <z...@ics.forth.gr> wrote:
>>>> 
>>>> Hello, 
>>>> 
>>>> I am using the Mesos Containerizer with Docker Images. The problem is that 
>>>> whenever a container exits my task gets TASK_FAILED because the container 
>>>> exits with ‘1’.
>>>> My docker file invokes a shell script via CMD /script.sh.
>>>> 
>>>> My protobuff can be found below:
>>>> https://pastebin.com/1agjAFdm
>>>> 
>>>> 
>>>> The agent log can be found here:
>>>> https://pastebin.com/Q6qndVuU
>>>> 
>>>> 
>>>> The stderr and stdout from the UI can be found here:
>>>> stdout: https://pastebin.com/SkDDUDDJ
>>>> 
>>>> stderr: https://pastebin.com/t2TFgQ4G
>>> /execute_blast.sh: line 4: /proc/sys/vm/drop_caches: Read-only file system
>>> 
>>> 
>>> Does your task exit when this happens?
>>> 
>>> 
> 
> -- 
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95
> 
> gpg key id: 4096R/326D8438  (keyring.debian.org <http://keyring.debian.org/>)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438



Re: Welcome James Peach as a new committer and PMC memeber!

2017-09-06 Thread Thodoris Zois
Congratulations James!! 

Thodoris 

> On 7 Sep 2017, at 01:06, Avinash Sridharan  wrote:
> 
> Congrats James !! Very well deserved.
>> On Wed, Sep 6, 2017 at 5:47 PM Benjamin Mahler  wrote:
>> 
>> Thanks for all that you've done so far for the project James!
>> 
>>> On Wed, Sep 6, 2017 at 2:08 PM, Yan Xu  wrote:
>>> 
>>> Hi Mesos devs and users,
>>> 
>>> Please welcome James Peach as a new Apache Mesos committer and PMC member.
>>> 
>>> James has been an active contributor to Mesos for over two years now. He
>>> has made many great contributions to the project which include XFS disk
>>> isolator, improvement to Linux capabilities support and IPC namespace
>>> isolator. He's super active on the mailing lists and slack channels, always
>>> eager to help folks in the community and he has been helping with a lot of
>>> Mesos reviews as well.
>>> 
>>> Here is his formal committer candidate checklist:
>>> 
>>> 
>>> https://docs.google.com/document/d/19G5zSxhrRBdS6GXn9KjCznjX3cp0mUbck6Jy1Hgn3RY/edit?usp=sharing
>>> 
>>> 
>>> Congrats James!
>>> 
>>> Yan
>>> 
>>> 



Re: TASK_FAILED - Mesos Container Images

2017-09-06 Thread Thodoris Zois
Yes you are right, this is the problem. I aso noticed it. Docker image has no 
privileges so script can't execute this command. Is there any way to use 
arguments in proto? Or i have to set command to the task? And set shell to 
true? 

Thodoris

> On 6 Sep 2017, at 18:26, James Peach <jor...@gmail.com> wrote:
> 
> 
>> On Sep 6, 2017, at 4:41 AM, Thodoris Zois <z...@ics.forth.gr> wrote:
>> 
>> Hello, 
>> 
>> I am using the Mesos Containerizer with Docker Images. The problem is that 
>> whenever a container exits my task gets TASK_FAILED because the container 
>> exits with ‘1’.
>> My docker file invokes a shell script via CMD /script.sh.
>> 
>> My protobuff can be found below:
>> https://pastebin.com/1agjAFdm
>> 
>> 
>> The agent log can be found here:
>> https://pastebin.com/Q6qndVuU
>> 
>> 
>> The stderr and stdout from the UI can be found here:
>> stdout: https://pastebin.com/SkDDUDDJ
>> 
>> stderr: https://pastebin.com/t2TFgQ4G
> 
> /execute_blast.sh: line 4: /proc/sys/vm/drop_caches: Read-only file system
> 
> 
> Does your task exit when this happens?
> 
> 



TASK_FAILED - Mesos Container Images

2017-09-06 Thread Thodoris Zois
Hello, 

I am using the Mesos Containerizer with Docker Images. The problem is that 
whenever a container exits my task gets TASK_FAILED because the container exits 
with ‘1’.
My docker file invokes a shell script via CMD /script.sh.

My protobuff can be found below:
https://pastebin.com/1agjAFdm 


The agent log can be found here:
https://pastebin.com/Q6qndVuU 


The stderr and stdout from the UI can be found here:
stdout: https://pastebin.com/SkDDUDDJ 

stderr: https://pastebin.com/t2TFgQ4G 


Any help would be appreciated,
Thank you very much!


Zois Theodoros

Mesos workloads

2017-08-10 Thread Thodoris Zois
Hello,

 I am planning on submitting a manuscript of my work later this year. I am 
trying to evaluate the results of my work on real workloads. Is there somebody 
that would like to help by giving workloads that his/her company use along with 
mesos? Or some traces so i can make my own workload? Sorry for notifying the 
whole channel. If somebody would like to help do not hesitate to send me pm.


Thank you,
Thodoris Zois

--
Institute of Computer Science, Greece
FORTH-ICS 


Re: Custom isolators - External container

2017-08-07 Thread Thodoris Zois
Hello, 

Thank you for your response, i will use the Mesos containerizer and run docker 
images as tasks, so i will use cgroups/cpu and cgroups/mem for isolation in 
order to get some results so i can compare them when in case i use my own 
isolators. Isolation module will work right? Because of the use of mesos 
containerizer..

Also, where i can find some workloads that are used along with mesos so i can 
evaluate better my results or traces so i can make my own workloads. What kind 
of tasks most people run with Mesos? 






> On 7 Aug 2017, at 21:16, Joseph Wu <jos...@mesosphere.io> wrote:
> 
> First off, the external containerizer was officially removed in Mesos 1.1.0 
> (it had been deprecated long before that release):
> https://issues.apache.org/jira/browse/MESOS-3370
> 
> ---
> 
> If you want to develop/deploy a new isolation method for Mesos, you should 
> first consider writing isolator modules (Mesos modules):
> https://github.com/apache/mesos/blob/master/include/mesos/slave/isolator.hpp
> 
> Isolator modules are only applicable for the Mesos containerizer, so if you 
> plan to run docker workloads, you can consider using built-in isolators 
> ("docker/runtime") that support running docker images in the Mesos 
> containerizer.
> 
> If you plan to use the Docker containerizer, your only choice is to develop a 
> custom executor to isolate tasks only within the same executor (docker will 
> take over isolating executors from each other).
> 
> ---
> 
> There are few benefits from running the Mesos agent inside a Docker container 
> and many pitfalls, so this practice is highly discouraged.  Instead, we 
> recommend running the Mesos agent directly via a supervisor (upstart, 
> systemd, etc.).  The agent itself is not containerized when run normally.
> 
>> On Sun, Aug 6, 2017 at 4:32 PM, Thodoris Zois <z...@ics.forth.gr> wrote:
>> Hello,
>> 
>> Is support of external containerizer removed from Mesos? Also, i have 
>> developed some isolators that i would like to use with Mesos. I found 3 ways 
>> to do that but i don't know what is the proper way and what are the 
>> advantages and disadvantages in each case.
>> 
>> The 1st one is as a Mesos module
>> 
>> The 2nd one is a custom executor
>> 
>> The 3rd one is the container image on agent.
>> 
>> What i am trying to do is to isolate docker tasks (images - one task per 
>> docker container) that run under the same agent with my own isolators.
>> 
>> What are the benefits of running agent in a big docker container and inside 
>> small docker containers as tasks?  If you don't run the agent under a big 
>> docker container  then by default is running under Mesos container while 
>> inside are running small docker containers with tasks? (Assume
>> that we don't run tasks under mesos container)
>> 
>> 
>> Thank you and sorry for the so many questions!
>> Thodoris
> 


Custom isolators - External container

2017-08-06 Thread Thodoris Zois
Hello, 

Is support of external containerizer removed from Mesos? Also, i have developed 
some isolators that i would like to use with Mesos. I found 3 ways to do that 
but i don't know what is the proper way and what are the advantages and 
disadvantages in each case. 

The 1st one is as a Mesos module

The 2nd one is a custom executor

The 3rd one is the container image on agent. 

What i am trying to do is to isolate docker tasks (images - one task per docker 
container) that run under the same agent with my own isolators.

What are the benefits of running agent in a big docker container and inside 
small docker containers as tasks?  If you don't run the agent under a big 
docker container  then by default is running under Mesos container while inside 
are running small docker containers with tasks? (Assume
that we don't run tasks under mesos container)


Thank you and sorry for the so many questions!
Thodoris 


Re: Docker image as an executor

2017-08-04 Thread Thodoris Zois
Thank you very much both of you!

Thodoris

> On 4 Aug 2017, at 17:47, Vinod Kone <vinodk...@gmail.com> wrote:
> 
> Setting container on task is the right thing to do for your use case. Each 
> task is run under a different executor and hence isolated. 
> 
> @vinodkone
> 
> On Aug 4, 2017, at 6:10 AM, Jan Schlicht <j...@mesosphere.io 
> <mailto:j...@mesosphere.io>> wrote:
> 
>> It depends on the executor how the `TaskInfo` is used. The default executor 
>> (i.e. `TaskInfo.executor` isn't set) will start a container per task -- as 
>> documented here: 
>> http://mesos.apache.org/documentation/latest/app-framework-development-guide/
>>  
>> <http://mesos.apache.org/documentation/latest/app-framework-development-guide/>.
>> 
>> Cheers,
>> Jan
>> 
>> On Fri, Aug 4, 2017 at 2:55 PM, Thodoris Zois <z...@ics.forth.gr 
>> <mailto:z...@ics.forth.gr>> wrote:
>> Hello Jan,
>> 
>> You are completely right.. If i set the docker image at taskInfo it is 
>> running properly. The reason that i wanted to add the docker image as an 
>> executor is because i would like to have one task per container. If i 
>> setContainer at taskInfo every task is going to run under the same 
>> container? Or each task will have its own container? I would like to provide 
>> an isolation between docker containers and tasks to my framework.
>> 
>> Thank you,
>> Thodoris
>> 
>>> On 4 Aug 2017, at 15:50, Jan Schlicht <j...@mesosphere.io 
>>> <mailto:j...@mesosphere.io>> wrote:
>>> 
>>> Hey Thodoris,
>>> 
>>> when you set a container for the executor (the `ExecutorInfo.container` 
>>> field), this container is expected to provide an executor, i.e. connects 
>>> and communicates with the Mesos agent that started the executor to run a 
>>> task. `echo hello world` doesn't do this, that's why you see the 
>>> EXECUTOR_TERMINATED message.
>>> 
>>> Cheers
>>> Jan
>>> 
>>> On Fri, Aug 4, 2017 at 1:44 PM, Thodoris Zois <z...@ics.forth.gr 
>>> <mailto:z...@ics.forth.gr>> wrote:
>>> Hello list,
>>> 
>>> I am developing my own framework and i got a problem with docker images.
>>> 
>>> I am creating a task that will have assigned a docker image as executor. 
>>> The purpose of doing this is because i want to have one task per docker 
>>> container.  I have set a TaskInfo with executor a `docker image`. In the 
>>> image i am just running `echo hello world!`. Every task that uses `Executor 
>>> docker image` is in state `TASK_FAILED with reason EXECUTOR_TERMINATED -> 
>>> Container terminated`. I don't get any status update message `TASK_RUNNING` 
>>> and the image is running properly. Even at the output of stdout i get the 
>>> message `Hello world!`. I think that the task is not getting in my 
>>> container. But it gets the proper executorID.
>>> 
>>> This is my master log: https://pastebin.com/5yB7KR28 
>>> <https://pastebin.com/5yB7KR28> <https://pastebin.com/5yB7KR28 
>>> <https://pastebin.com/5yB7KR28>>
>>> 
>>> This is my slave log: https://pastebin.com/sKPz949J 
>>> <https://pastebin.com/sKPz949J> <https://pastebin.com/sKPz949J 
>>> <https://pastebin.com/sKPz949J>>
>>> 
>>> And this is my protobuf: https://pastebin.com/YG3J7Fi0 
>>> <https://pastebin.com/YG3J7Fi0> <https://pastebin.com/YG3J7Fi0 
>>> <https://pastebin.com/YG3J7Fi0>>
>>> 
>>> 
>>> Any help would be appreciated!
>>> 
>>> Thank you,
>>> Thodoris
>>> 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Jan Schlicht
>>> Distributed Systems Engineer, Mesosphere
>> 
>> 
>> 
>> 
>> -- 
>> Jan Schlicht
>> Distributed Systems Engineer, Mesosphere



Re: Docker image as an executor

2017-08-04 Thread Thodoris Zois
Hello Jan,

You are completely right.. If i set the docker image at taskInfo it is running 
properly. The reason that i wanted to add the docker image as an executor is 
because i would like to have one task per container. If i setContainer at 
taskInfo every task is going to run under the same container? Or each task will 
have its own container? I would like to provide an isolation between docker 
containers and tasks to my framework.

Thank you,
Thodoris

> On 4 Aug 2017, at 15:50, Jan Schlicht <j...@mesosphere.io> wrote:
> 
> Hey Thodoris,
> 
> when you set a container for the executor (the `ExecutorInfo.container` 
> field), this container is expected to provide an executor, i.e. connects and 
> communicates with the Mesos agent that started the executor to run a task. 
> `echo hello world` doesn't do this, that's why you see the 
> EXECUTOR_TERMINATED message.
> 
> Cheers
> Jan
> 
> On Fri, Aug 4, 2017 at 1:44 PM, Thodoris Zois <z...@ics.forth.gr 
> <mailto:z...@ics.forth.gr>> wrote:
> Hello list,
> 
> I am developing my own framework and i got a problem with docker images.
> 
> I am creating a task that will have assigned a docker image as executor. The 
> purpose of doing this is because i want to have one task per docker 
> container.  I have set a TaskInfo with executor a `docker image`. In the 
> image i am just running `echo hello world!`. Every task that uses `Executor 
> docker image` is in state `TASK_FAILED with reason EXECUTOR_TERMINATED -> 
> Container terminated`. I don't get any status update message `TASK_RUNNING` 
> and the image is running properly. Even at the output of stdout i get the 
> message `Hello world!`. I think that the task is not getting in my container. 
> But it gets the proper executorID.
> 
> This is my master log: https://pastebin.com/5yB7KR28 
> <https://pastebin.com/5yB7KR28> <https://pastebin.com/5yB7KR28 
> <https://pastebin.com/5yB7KR28>>
> 
> This is my slave log: https://pastebin.com/sKPz949J 
> <https://pastebin.com/sKPz949J> <https://pastebin.com/sKPz949J 
> <https://pastebin.com/sKPz949J>>
> 
> And this is my protobuf: https://pastebin.com/YG3J7Fi0 
> <https://pastebin.com/YG3J7Fi0> <https://pastebin.com/YG3J7Fi0 
> <https://pastebin.com/YG3J7Fi0>>
> 
> 
> Any help would be appreciated!
> 
> Thank you,
> Thodoris
> 
> 
> 
> 
> 
> -- 
> Jan Schlicht
> Distributed Systems Engineer, Mesosphere



Docker image as an executor

2017-08-04 Thread Thodoris Zois
Hello list,

I am developing my own framework and i got a problem with docker images.

I am creating a task that will have assigned a docker image as executor. The 
purpose of doing this is because i want to have one task per docker container.  
I have set a TaskInfo with executor a `docker image`. In the image i am just 
running `echo hello world!`. Every task that uses `Executor docker image` is in 
state `TASK_FAILED with reason EXECUTOR_TERMINATED -> Container terminated`. I 
don't get any status update message `TASK_RUNNING` and the image is running 
properly. Even at the output of stdout i get the message `Hello world!`. I 
think that the task is not getting in my container. But it gets the proper 
executorID.

This is my master log: https://pastebin.com/5yB7KR28 


This is my slave log: https://pastebin.com/sKPz949J 


And this is my protobuf: https://pastebin.com/YG3J7Fi0 



Any help would be appreciated!

Thank you, 
Thodoris




Re: Containerizers & Executors

2017-07-31 Thread Thodoris Zois
Hello,

Thank you for your response, but i am still a bit confused. 

What Mesos containerizer does? What is the point of a containerizer? 

Can commandExecutor launch a group of tasks? 

And finally is there any default executor used in tests so i can see an 
example? I guess that custom executor is for example TestExecutor.java. 
CommandExecutor is specified on taskInfo.command.commandInfo. So what about the 
default if i want to set? 

Thodoris



> On 31 Jul 2017, at 04:46, Vinod Kone  wrote:
> 
> See my answers inline.
> 
>> 
>> 1. Mesos Containerizer
>> - posix isolators
>> - cgroups isolators
> 
> Mesos container also allows you to use custom isolators. 
> 
>  
>> 2. Docker containerizer
>> - docker isolators
> 
> Docker containerizer doesn't have a concept of isolator(s).
> 
>  
>> 3. Custom containerizer
>> - my isolators
> 
> It is up to the custom containerizer how it wants to do containerization; it 
> could've have a concept of isolator or not.
>  
> 
>> - Executors:
>> Generally: Each executor has the minimum resources assigned by default 
>> (0.01 CPU & 32MB MEM)
>>Executor expands its resources when a task is assigned 
>> (executor default resources + task resources)
> 
> Only the built-in "default" executor needs to have a minimum amount of 
> resources. Other built-in executors and custom executors can technically have 
> zero resources.
> 
>  
>> 1. Mesos commandExecutor
>> - run shell commands or docker
>> - Each executor is a container that can have only one task to 
>> execute, you can't specify group of tasks
>> - Isolation between executors/containers so isolation between tasks 
>> because each task runs in one container
> 
> Not that the executor that runs shell commands is called the "command" 
> executor (run by mesos containerizer), whereas the one that runs docker 
> images is called "docker" executor (run by docker containerizer).
>  
>  
>> 2. Mesos defaultExecutor
>> - can run shell commands or a custom executor file e.g 
>> TestExecutor.java (from tests)
>> - can execute one task per executor/container or multiple tasks (1 
>> group).
>> - No resource isolation between tasks of the same container
> 
> "default" executor is another built-in executor. it can run a group of tasks. 
> it does not run any other (custom) executor.
> 
>  
>> 3. Custom Executor
>> - ?
> 
> you could write a custom executor that can run a single task or group of 
> tasks. totally up to you.
> 
>  
>> So, i guess i can use one offer to run some tasks on the same agent with 
>> commandExecutor or with defaultExecutor….
>> But how would somebody specify if the offer corresponds to one agent or 
>> multiple agents?
> 
> Each offer has an 'AgentId' which corresponds to one agent.
> 
> HTH,
> Vinod 
> 


Containerizers & Executors

2017-07-28 Thread Thodoris Zois
Hello, 

I am trying to understand some basic components of Mesos and i have made some 
notes corresponding to my conclusion. If somebody could check this out and 
review them, or can provide me some more information about the specific topic i 
would be thankful. Below you will find my notes.


- Containerizers: 
1. Mesos Containerizer
- posix isolators 
- cgroups isolators

2. Docker containerizer
- docker isolators

3. Custom containerizer
- my isolators


- Executors:
Generally: Each executor has the minimum resources assigned by default 
(0.01 CPU & 32MB MEM)
   Executor expands its resources when a task is assigned (executor 
default resources + task resources)

1. Mesos commandExecutor
- run shell commands or docker
- Each executor is a container that can have only one task to execute, 
you can't specify group of tasks
- Isolation between executors/containers so isolation between tasks 
because each task runs in one container

2. Mesos defaultExecutor
- can run shell commands or a custom executor file e.g 
TestExecutor.java (from tests)
- can execute one task per executor/container or multiple tasks (1 
group).
- No resource isolation between tasks of the same container

3. Custom Executor
- ?


So, i guess i can use one offer to run some tasks on the same agent with 
commandExecutor or with defaultExecutor….
But how would somebody specify if the offer corresponds to one agent or 
multiple agents? 

Thank you,
Thodoris



Re: Framework change role

2017-07-07 Thread Thodoris Zois
Thank you very much! I will read everything! 


> On 5 Jul 2017, at 19:04, Gabriel Hartmann <gabr...@mesosphere.io> wrote:
> 
> * Maybe have a look at the DefaultScheduler there as well, if you're just 
> trying to get things building and off the ground 
> (https://github.com/mesosphere/dcos-commons/blob/master/sdk/scheduler/src/main/java/com/mesosphere/sdk/scheduler/DefaultScheduler.java#L53
>  
> <https://github.com/mesosphere/dcos-commons/blob/master/sdk/scheduler/src/main/java/com/mesosphere/sdk/scheduler/DefaultScheduler.java#L53>).
> 
> On Wed, Jul 5, 2017 at 9:03 AM Gabriel Hartmann <gabr...@mesosphere.io 
> <mailto:gabr...@mesosphere.io>> wrote:
> You may register in multiple roles simultaneously.  Register with the 
> MULTI_ROLE capability 
> (https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L345 
> <https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L345>).
>   Our frameworks all register with a role, but never the role '*'.  We get 
> Offers from all agents that don't have statically partitioned resources and 
> do dynamic reservation into one of our registered roles.  I don't know the 
> purpose or goals of your project, but some example Java based frameworks are 
> available here: https://github.com/mesosphere/dcos-commons 
> <https://github.com/mesosphere/dcos-commons>.
> 
> We use multiple roles when appropriate here: 
> https://github.com/mesosphere/dcos-commons/blob/master/sdk/scheduler/src/main/java/com/mesosphere/sdk/specification/DefaultService.java#L200
>  
> <https://github.com/mesosphere/dcos-commons/blob/master/sdk/scheduler/src/main/java/com/mesosphere/sdk/specification/DefaultService.java#L200>
> 
> Maybe have a look at the DefaultScheduler for an implementation of a look at 
> the DefaultScheduler there as well, if you're just trying to get things 
> building and off the ground 
> (https://github.com/mesosphere/dcos-commons/blob/master/sdk/scheduler/src/main/java/com/mesosphere/sdk/scheduler/DefaultScheduler.java#L53
>  
> <https://github.com/mesosphere/dcos-commons/blob/master/sdk/scheduler/src/main/java/com/mesosphere/sdk/scheduler/DefaultScheduler.java#L53>).
> 
> 
> 
> On Wed, Jul 5, 2017 at 3:13 AM James Peach <jor...@gmail.com 
> <mailto:jor...@gmail.com>> wrote:
> 
> > On Jul 5, 2017, at 2:39 AM, Thodoris Zois <z...@ics.forth.gr 
> > <mailto:z...@ics.forth.gr>> wrote:
> >
> > Ok, probably you are right but what you don’t understand is that i am a 
> > completely newbie and i see such systems for the first time. It’s about a 
> > university project that i am working on in order to get my bachelor degree. 
> > I don’t really know the proper way to express what i want to like you do. I 
> > don’t know how to connect 2 frameworks or something, i don’t even know how 
> > to make my own framework and compile it with the dependencies that Mesos 
> > needs. I am just writting on TestFramework.java and TestExecutor.java and 
> > that’s all. The reason that i want to run everything from the same 
> > framework (instance) is because keep a TreeMap with some info that i don’t 
> > want to lose if i terminate the Schedulers driver. So if i start up a new 
> > framework, TreeMap is gone. Forget about the tasks, i can use the same 
> > TestExecutor.java for every scheduler.
> 
> Note that you can start multiple Scheduler drivers within the same server. 
> Each one can register with Mesos as a separate framework.
> 
> > What i want to achieve is to get offers from a specific agent with role 
> > “SB” and run 10 tasks (i give also to my Framework role “SB”), then i store 
> > those information (which are actually TaskInfo in Schedulers TreeMap). 
> > After that i would like to change the role of my Framework to “*(default)”. 
> > Then i will get offer from an agent that has the default role and i will 
> > still have the info in my TreeMap because scheduler instance didn’t stop.
> 
> If you start multiple Scheduler drivers, they can all share a TreeMap because 
> they are running in the same Java process.
> 
> > That’s all. My problem is that i don’t know how to change the role of the 
> > Framework without losing that TreeMap, and also how to set it with version 
> > 1.3.0.
> >
> > Hope that everybody understands now….
> > Thank you, and i am really sorry for the spam
> >
> >
> >> On 5 Jul 2017, at 12:24, James Peach <jor...@gmail.com 
> >> <mailto:jor...@gmail.com>> wrote:
> >>
> >>
> >>> On Jul 5, 2017, at 12:54 AM, Thodoris Zois <z...@ics.forth.gr 
> >>> <mailto:z...@ics.forth.gr>> wrot

Re: Framework change role

2017-07-05 Thread Thodoris Zois
Ok, probably you are right but what you don’t understand is that i am a 
completely newbie and i see such systems for the first time. It’s about a 
university project that i am working on in order to get my bachelor degree. I 
don’t really know the proper way to express what i want to like you do. I don’t 
know how to connect 2 frameworks or something, i don’t even know how to make my 
own framework and compile it with the dependencies that Mesos needs. I am just 
writting on TestFramework.java and TestExecutor.java and that’s all. The reason 
that i want to run everything from the same framework (instance) is because 
keep a TreeMap with some info that i don’t want to lose if i terminate the 
Schedulers driver. So if i start up a new framework, TreeMap is gone. Forget 
about the tasks, i can use the same TestExecutor.java for every scheduler. 

What i want to achieve is to get offers from a specific agent with role “SB” 
and run 10 tasks (i give also to my Framework role “SB”), then i store those 
information (which are actually TaskInfo in Schedulers TreeMap). After that i 
would like to change the role of my Framework to “*(default)”. Then i will get 
offer from an agent that has the default role and i will still have the info in 
my TreeMap because scheduler instance didn’t stop.

That’s all. My problem is that i don’t know how to change the role of the 
Framework without losing that TreeMap, and also how to set it with version 
1.3.0.  

Hope that everybody understands now….
Thank you, and i am really sorry for the spam


> On 5 Jul 2017, at 12:24, James Peach <jor...@gmail.com> wrote:
> 
> 
>> On Jul 5, 2017, at 12:54 AM, Thodoris Zois <z...@ics.forth.gr> wrote:
>> 
>> Hi, 
>> 
>> No, i would like my framework to be offered resources from agent with role 
>> (e.g: thz) and after running the specific tasks change its role to (*) in 
>> order to get offers from different agents, but it will run the same tasks 
>> because i am never terminating the scheduler driver (that’s what i want to).
> 
> As I suggested on Slack, I still think the most obvious way to implement this 
> is to connect 2 frameworks, 1 for each role. Just co-ordinate internally to 
> accept the offers you want in the right sequence. From your description, 
> there's no requirement for this to be done in 1 framework.
> 
> I don't really follow what you mean by "run the same tasks". You can run new 
> instances of the same task (from whatever framework you like); you can also 
> send new tasks to an existing executor (from the same framework).
> 
>> Don’t try to find logic, it’s not for a company or something :)
> 
> I think that for people to help you, they need to be able to understand the 
> logic of what you want to achieve and why :)
> 
>> 
>> Thank you,
>> Thodoris
>> 
>>> On 5 Jul 2017, at 05:36, Jay Guo <guojiannan1...@gmail.com> wrote:
>>> 
>>> Hi Thodoris,
>>> 
>>> If I understand correctly, you would like your framework to receive offers 
>>> from both 'role' and '*', so resources reserved to 'role' on particular 
>>> agent could be reliably supplied to the framework? Isn't it sufficient to 
>>> start your framework with multiple roles, 'role' & '*'? You need to enable 
>>> the capability.
>>> 
>>> - J
>>> 
>>> On Wed, Jul 5, 2017 at 7:28 AM, Thodoris Zois <z...@ics.forth.gr> wrote:
>>> I have built a Framework in Java that is running certain tasks. I would 
>>> like to run those tasks on a specific agent. I have set a role to the 
>>> Framework and used flags upon starting of the agent. Till here everything 
>>> is good. When framework has run tasks successfully i am not terminating it. 
>>> I would like to change its role to default (*) and get offered resources 
>>> from master that correspond to that role and it will run again the same 
>>> amount of tasks (and the same tasks) because i never terminated (and i 
>>> don't want to terminate its instance because i keep some mesos metrics to a 
>>> static TreeMap). That's all.. I just wanted somebody to explain me exactly 
>>> how it works and what i have to do because everything i have tried today 
>>> fails, and seems i can't find useful info on the Internet about this. 
>>> 
>>> Thank you!
>>> 
>>> On 4 Jul 2017, at 21:00, Michael Park <mp...@apache.org> wrote:
>>> 
>>>> What is it that you need help with?
>>>> 
>>>> On Tue, Jul 4, 2017 at 11:12 AM Thodoris Zois <z...@ics.forth.gr> wrote:
>>>> Hello list,
>>>> 
>>>> Is anybody available to help me with the new feature of 1.3.0 version, 
>>>> that a framework can modify its role?
>>>> 
>>>> Thank you
>>> 
>> 
> 



Re: Framework change role

2017-07-05 Thread Thodoris Zois
Hi, 

No, i would like my framework to be offered resources from agent with role 
(e.g: thz) and after running the specific tasks change its role to (*) in order 
to get offers from different agents, but it will run the same tasks because i 
am never terminating the scheduler driver (that’s what i want to). Don’t try to 
find logic, it’s not for a company or something :)

Thank you,
Thodoris

> On 5 Jul 2017, at 05:36, Jay Guo <guojiannan1...@gmail.com> wrote:
> 
> Hi Thodoris,
> 
> If I understand correctly, you would like your framework to receive offers 
> from both 'role' and '*', so resources reserved to 'role' on particular agent 
> could be reliably supplied to the framework? Isn't it sufficient to start 
> your framework with multiple roles, 'role' & '*'? You need to enable the 
> capability.
> 
> - J
> 
> On Wed, Jul 5, 2017 at 7:28 AM, Thodoris Zois <z...@ics.forth.gr 
> <mailto:z...@ics.forth.gr>> wrote:
> I have built a Framework in Java that is running certain tasks. I would like 
> to run those tasks on a specific agent. I have set a role to the Framework 
> and used flags upon starting of the agent. Till here everything is good. When 
> framework has run tasks successfully i am not terminating it. I would like to 
> change its role to default (*) and get offered resources from master that 
> correspond to that role and it will run again the same amount of tasks (and 
> the same tasks) because i never terminated (and i don't want to terminate its 
> instance because i keep some mesos metrics to a static TreeMap). That's all.. 
> I just wanted somebody to explain me exactly how it works and what i have to 
> do because everything i have tried today fails, and seems i can't find useful 
> info on the Internet about this. 
> 
> Thank you!
> 
> On 4 Jul 2017, at 21:00, Michael Park <mp...@apache.org 
> <mailto:mp...@apache.org>> wrote:
> 
>> What is it that you need help with?
>> 
>> On Tue, Jul 4, 2017 at 11:12 AM Thodoris Zois <z...@ics.forth.gr 
>> <mailto:z...@ics.forth.gr>> wrote:
>> Hello list,
>> 
>> Is anybody available to help me with the new feature of 1.3.0 version, that 
>> a framework can modify its role?
>> 
>> Thank you
> 



Re: Framework change role

2017-07-04 Thread Thodoris Zois
I have built a Framework in Java that is running certain tasks. I would like to 
run those tasks on a specific agent. I have set a role to the Framework and 
used flags upon starting of the agent. Till here everything is good. When 
framework has run tasks successfully i am not terminating it. I would like to 
change its role to default (*) and get offered resources from master that 
correspond to that role and it will run again the same amount of tasks (and the 
same tasks) because i never terminated (and i don't want to terminate its 
instance because i keep some mesos metrics to a static TreeMap). That's all.. I 
just wanted somebody to explain me exactly how it works and what i have to do 
because everything i have tried today fails, and seems i can't find useful info 
on the Internet about this. 

Thank you!

> On 4 Jul 2017, at 21:00, Michael Park <mp...@apache.org> wrote:
> 
> What is it that you need help with?
> 
>> On Tue, Jul 4, 2017 at 11:12 AM Thodoris Zois <z...@ics.forth.gr> wrote:
>> Hello list,
>> 
>> Is anybody available to help me with the new feature of 1.3.0 version, that 
>> a framework can modify its role?
>> 
>> Thank you


Framework change role

2017-07-04 Thread Thodoris Zois
Hello list,

Is anybody available to help me with the new feature of 1.3.0 version, that a 
framework can modify its role? 

Thank you

Re: Mesos-Metrics per task

2017-06-30 Thread Thodoris Zois
From the resources that Mesos allocated for a specific task, i would like to 
get somehow how much the task truly used with percentage. Maybe it wasn’t clear 
before, that’s why i am mentioning it. At Mesos UI for some reason i don’t see 
enough info for tasks etc. They are all the time empty….



> On 30 Jun 2017, at 02:38, James Peach <jor...@gmail.com> wrote:
> 
> 
>> On Jun 29, 2017, at 3:53 PM, Thodoris Zois <z...@ics.forth.gr> wrote:
>> 
>> Hello, i would like to get some metrics per task. E.g memory/cpu usage is 
>> there any way? 
>> 
>> Thank you! 
> 
> You can use the GET_CONTAINERS agent API call 
> <http://mesos.apache.org/documentation/latest/operator-http-api/> to get 
> resource usage for a container, then match up the container to a task by 
> using other master and agent APIs to match the framework ID and executor ID.
> 
> J



Mesos-Metrics per task

2017-06-29 Thread Thodoris Zois
Hello, i would like to get some metrics per task. E.g memory/cpu usage is there 
any way? 

Thank you! 


Compiling my own framework

2017-06-21 Thread Thodoris Zois
Hello list,

I have created my own framework in Java and i don’t know how to compile it. Any 
help with some instructions would be really appreciated. Right now i am just 
replacing the code in TestFramework.java so i can run my code after compiling 
it with ‘make tests’. But compile takes time, so i wonder how i can make this 
procedure fast!

Thank you,
Thodoris