Re: 2 Mesos Slaves in same machine
Hi, Is it still an issue with the below config as we have cgroup setting over there? Do i need to use both "--resources" && "--cgroups_root" setting for running 2 slaves in the same node. Or, either one of them should do. Also, is there any setting to be done at the OS( cgroup ) level for running 2 slaves. Anyway, since it is not a recommended to run 2 slaves on the same node, I am not going with that option. Just trying to understand here ... Thanks On Thu, Mar 8, 2018 at 11:02 AM, Baskar Sikkayan wrote: > Hi, > > This is our docker compose for starting mesos slave. > > mesos-slave1: > > image: docker.com/mesos/mesos-slave:0.28.2 > > depends_on: [ mesos-master ] > > restart: always > > privileged: true > > network_mode: host > > volumes: > > - ~/mesos-data/slave-1:/tmp/mesos > > - /cgroup:/sys/fs/cgroup > > - /var/run/docker.sock:/var/run/docker.sock > > environment: > > MESOS_CONTAINERIZERS: docker > > MESOS_MASTER: "%mesos.zk%" > > MESOS_PORT: 5051 > > MESOS_LOGGING_LEVEL: WARNING > > MESOS_INITIALIZE_DRIVER_LOGGING: "false" > > MESOS_GC_DELAY: "%mesos.gc.delay%" > > MESOS_AUTHENTICATE_HTTP: "true" > > PRINCIPAL: "%mesos.principal%" > > SECRET: "%mesos.secret%" > > > We are running mesos version 0.28.2. I know, it is pretty old. But no > option to upgrade in the near future. We use Mesos containerizers. > > Is this still an issue with 2 mesos salves on the same node? > > > Thanks > > On Wed, Mar 7, 2018 at 10:11 PM, Jie Yu wrote: > >> Running 2 agents on the same server will send the same offer twice( same >>> offer by each agent ) and there are chances that resources will be over >>> utilized by accepting the same offer twice by the framework. >> >> >> If you didn't specify `--resources` flags on the agent, then yes. >> >> May I know the settings( cgroup ) required to run 2 slaves in the same >>> server and How does it behave? >> >> >> Check out `--cgroups_root` flag in https://github.com/apache/m >> esos/blob/master/docs/configuration/agent.md >> >> This flag only applies to MesosContainerizer (can be used to launch >> Docker containers). >> >> - Jie >> >> >> On Wed, Mar 7, 2018 at 10:06 PM, Baskar Sikkayan >> wrote: >> >>> 1. Is my understanding correct? >>> >>> Running 2 agents on the same server will send the same offer twice( same >>> offer by each agent ) and there are chances that resources will be over >>> utilized by accepting the same offer twice by the framework. >>> >>> 2. May I know the settings( cgroup ) required to run 2 slaves in the >>> same server and How does it behave? >>> >>> On Wed, Mar 7, 2018 at 10:03 PM, Jie Yu wrote: >>> Running Docker containers won't work properly because restarting one agent will cause Docker containers managed by the other agent to be deleted. On Wed, Mar 7, 2018 at 9:58 PM, Baskar Sikkayan wrote: > We dont have any isolation setting. Looks like 2 slaves are sending > too many offers( same resource offers by 2 agents ) and hence servers gets > overloaded by too many docker jobs and docker becomes unresponsive. Even > "docker ps" not working in this case and docker meta files gets corrupted > on server reboot. Not sure why docker becomes unresponsive. Is running 2 > slaves are the main reason for this issue? > > On Wed, Mar 7, 2018 at 9:52 PM, Jie Yu wrote: > >> It depends on your isolation setting (mainly cgroup, or any node >> level resources). In general, we don't recommend folks use multiple >> agents >> on a node. >> >> It's possible to make it work by setting `cgroup_root` separately for >> MesosContainerizer. For DockerContainerizer, currently, we hard code >> `DOCKER_NAME_PREFIX`, making it not possible to use two agents on a node >> properly. >> >> - Jie >> >> On Wed, Mar 7, 2018 at 9:48 PM, Baskar Sikkayan > > wrote: >> >>> Hi, >>> >>> We are running 3 node mesos cluster and on each node running 1 >>> mesos master and 2 mesos slaves. Is this a good practice? >>> >>> Will it be a problem running 2 slaves as it might offer too much and >>> will get overloaded. >>> >>> >>> Thanks, >>> Baskar.S >>> >> >> > >>> >> >
Re: 2 Mesos Slaves in same machine
Hi, This is our docker compose for starting mesos slave. mesos-slave1: image: docker.com/mesos/mesos-slave:0.28.2 depends_on: [ mesos-master ] restart: always privileged: true network_mode: host volumes: - ~/mesos-data/slave-1:/tmp/mesos - /cgroup:/sys/fs/cgroup - /var/run/docker.sock:/var/run/docker.sock environment: MESOS_CONTAINERIZERS: docker MESOS_MASTER: "%mesos.zk%" MESOS_PORT: 5051 MESOS_LOGGING_LEVEL: WARNING MESOS_INITIALIZE_DRIVER_LOGGING: "false" MESOS_GC_DELAY: "%mesos.gc.delay%" MESOS_AUTHENTICATE_HTTP: "true" PRINCIPAL: "%mesos.principal%" SECRET: "%mesos.secret%" We are running mesos version 0.28.2. I know, it is pretty old. But no option to upgrade in the near future. We use Mesos containerizers. Is this still an issue with 2 mesos salves on the same node? Thanks On Wed, Mar 7, 2018 at 10:11 PM, Jie Yu wrote: > Running 2 agents on the same server will send the same offer twice( same >> offer by each agent ) and there are chances that resources will be over >> utilized by accepting the same offer twice by the framework. > > > If you didn't specify `--resources` flags on the agent, then yes. > > May I know the settings( cgroup ) required to run 2 slaves in the same >> server and How does it behave? > > > Check out `--cgroups_root` flag in https://github.com/apache/ > mesos/blob/master/docs/configuration/agent.md > > This flag only applies to MesosContainerizer (can be used to launch Docker > containers). > > - Jie > > > On Wed, Mar 7, 2018 at 10:06 PM, Baskar Sikkayan > wrote: > >> 1. Is my understanding correct? >> >> Running 2 agents on the same server will send the same offer twice( same >> offer by each agent ) and there are chances that resources will be over >> utilized by accepting the same offer twice by the framework. >> >> 2. May I know the settings( cgroup ) required to run 2 slaves in the same >> server and How does it behave? >> >> On Wed, Mar 7, 2018 at 10:03 PM, Jie Yu wrote: >> >>> Running Docker containers won't work properly because restarting one >>> agent will cause Docker containers managed by the other agent to be deleted. >>> >>> On Wed, Mar 7, 2018 at 9:58 PM, Baskar Sikkayan >>> wrote: >>> We dont have any isolation setting. Looks like 2 slaves are sending too many offers( same resource offers by 2 agents ) and hence servers gets overloaded by too many docker jobs and docker becomes unresponsive. Even "docker ps" not working in this case and docker meta files gets corrupted on server reboot. Not sure why docker becomes unresponsive. Is running 2 slaves are the main reason for this issue? On Wed, Mar 7, 2018 at 9:52 PM, Jie Yu wrote: > It depends on your isolation setting (mainly cgroup, or any node level > resources). In general, we don't recommend folks use multiple agents on a > node. > > It's possible to make it work by setting `cgroup_root` separately for > MesosContainerizer. For DockerContainerizer, currently, we hard code > `DOCKER_NAME_PREFIX`, making it not possible to use two agents on a node > properly. > > - Jie > > On Wed, Mar 7, 2018 at 9:48 PM, Baskar Sikkayan > wrote: > >> Hi, >> >> We are running 3 node mesos cluster and on each node running 1 >> mesos master and 2 mesos slaves. Is this a good practice? >> >> Will it be a problem running 2 slaves as it might offer too much and >> will get overloaded. >> >> >> Thanks, >> Baskar.S >> > > >>> >> >
Re: 2 Mesos Slaves in same machine
> > Running 2 agents on the same server will send the same offer twice( same > offer by each agent ) and there are chances that resources will be over > utilized by accepting the same offer twice by the framework. If you didn't specify `--resources` flags on the agent, then yes. May I know the settings( cgroup ) required to run 2 slaves in the same > server and How does it behave? Check out `--cgroups_root` flag in https://github.com/apache/mesos/blob/master/docs/configuration/agent.md This flag only applies to MesosContainerizer (can be used to launch Docker containers). - Jie On Wed, Mar 7, 2018 at 10:06 PM, Baskar Sikkayan wrote: > 1. Is my understanding correct? > > Running 2 agents on the same server will send the same offer twice( same > offer by each agent ) and there are chances that resources will be over > utilized by accepting the same offer twice by the framework. > > 2. May I know the settings( cgroup ) required to run 2 slaves in the same > server and How does it behave? > > On Wed, Mar 7, 2018 at 10:03 PM, Jie Yu wrote: > >> Running Docker containers won't work properly because restarting one >> agent will cause Docker containers managed by the other agent to be deleted. >> >> On Wed, Mar 7, 2018 at 9:58 PM, Baskar Sikkayan >> wrote: >> >>> We dont have any isolation setting. Looks like 2 slaves are sending too >>> many offers( same resource offers by 2 agents ) and hence servers gets >>> overloaded by too many docker jobs and docker becomes unresponsive. Even >>> "docker ps" not working in this case and docker meta files gets corrupted >>> on server reboot. Not sure why docker becomes unresponsive. Is running 2 >>> slaves are the main reason for this issue? >>> >>> On Wed, Mar 7, 2018 at 9:52 PM, Jie Yu wrote: >>> It depends on your isolation setting (mainly cgroup, or any node level resources). In general, we don't recommend folks use multiple agents on a node. It's possible to make it work by setting `cgroup_root` separately for MesosContainerizer. For DockerContainerizer, currently, we hard code `DOCKER_NAME_PREFIX`, making it not possible to use two agents on a node properly. - Jie On Wed, Mar 7, 2018 at 9:48 PM, Baskar Sikkayan wrote: > Hi, > > We are running 3 node mesos cluster and on each node running 1 mesos > master and 2 mesos slaves. Is this a good practice? > > Will it be a problem running 2 slaves as it might offer too much and > will get overloaded. > > > Thanks, > Baskar.S > >>> >> >
Re: 2 Mesos Slaves in same machine
1. Is my understanding correct? Running 2 agents on the same server will send the same offer twice( same offer by each agent ) and there are chances that resources will be over utilized by accepting the same offer twice by the framework. 2. May I know the settings( cgroup ) required to run 2 slaves in the same server and How does it behave? On Wed, Mar 7, 2018 at 10:03 PM, Jie Yu wrote: > Running Docker containers won't work properly because restarting one agent > will cause Docker containers managed by the other agent to be deleted. > > On Wed, Mar 7, 2018 at 9:58 PM, Baskar Sikkayan > wrote: > >> We dont have any isolation setting. Looks like 2 slaves are sending too >> many offers( same resource offers by 2 agents ) and hence servers gets >> overloaded by too many docker jobs and docker becomes unresponsive. Even >> "docker ps" not working in this case and docker meta files gets corrupted >> on server reboot. Not sure why docker becomes unresponsive. Is running 2 >> slaves are the main reason for this issue? >> >> On Wed, Mar 7, 2018 at 9:52 PM, Jie Yu wrote: >> >>> It depends on your isolation setting (mainly cgroup, or any node level >>> resources). In general, we don't recommend folks use multiple agents on a >>> node. >>> >>> It's possible to make it work by setting `cgroup_root` separately for >>> MesosContainerizer. For DockerContainerizer, currently, we hard code >>> `DOCKER_NAME_PREFIX`, making it not possible to use two agents on a node >>> properly. >>> >>> - Jie >>> >>> On Wed, Mar 7, 2018 at 9:48 PM, Baskar Sikkayan >>> wrote: >>> Hi, We are running 3 node mesos cluster and on each node running 1 mesos master and 2 mesos slaves. Is this a good practice? Will it be a problem running 2 slaves as it might offer too much and will get overloaded. Thanks, Baskar.S >>> >>> >> >
Re: 2 Mesos Slaves in same machine
Running Docker containers won't work properly because restarting one agent will cause Docker containers managed by the other agent to be deleted. On Wed, Mar 7, 2018 at 9:58 PM, Baskar Sikkayan wrote: > We dont have any isolation setting. Looks like 2 slaves are sending too > many offers( same resource offers by 2 agents ) and hence servers gets > overloaded by too many docker jobs and docker becomes unresponsive. Even > "docker ps" not working in this case and docker meta files gets corrupted > on server reboot. Not sure why docker becomes unresponsive. Is running 2 > slaves are the main reason for this issue? > > On Wed, Mar 7, 2018 at 9:52 PM, Jie Yu wrote: > >> It depends on your isolation setting (mainly cgroup, or any node level >> resources). In general, we don't recommend folks use multiple agents on a >> node. >> >> It's possible to make it work by setting `cgroup_root` separately for >> MesosContainerizer. For DockerContainerizer, currently, we hard code >> `DOCKER_NAME_PREFIX`, making it not possible to use two agents on a node >> properly. >> >> - Jie >> >> On Wed, Mar 7, 2018 at 9:48 PM, Baskar Sikkayan >> wrote: >> >>> Hi, >>> >>> We are running 3 node mesos cluster and on each node running 1 mesos >>> master and 2 mesos slaves. Is this a good practice? >>> >>> Will it be a problem running 2 slaves as it might offer too much and >>> will get overloaded. >>> >>> >>> Thanks, >>> Baskar.S >>> >> >> >
Re: 2 Mesos Slaves in same machine
We dont have any isolation setting. Looks like 2 slaves are sending too many offers( same resource offers by 2 agents ) and hence servers gets overloaded by too many docker jobs and docker becomes unresponsive. Even "docker ps" not working in this case and docker meta files gets corrupted on server reboot. Not sure why docker becomes unresponsive. Is running 2 slaves are the main reason for this issue? On Wed, Mar 7, 2018 at 9:52 PM, Jie Yu wrote: > It depends on your isolation setting (mainly cgroup, or any node level > resources). In general, we don't recommend folks use multiple agents on a > node. > > It's possible to make it work by setting `cgroup_root` separately for > MesosContainerizer. For DockerContainerizer, currently, we hard code > `DOCKER_NAME_PREFIX`, making it not possible to use two agents on a node > properly. > > - Jie > > On Wed, Mar 7, 2018 at 9:48 PM, Baskar Sikkayan > wrote: > >> Hi, >> >> We are running 3 node mesos cluster and on each node running 1 mesos >> master and 2 mesos slaves. Is this a good practice? >> >> Will it be a problem running 2 slaves as it might offer too much and will >> get overloaded. >> >> >> Thanks, >> Baskar.S >> > >
Re: 2 Mesos Slaves in same machine
It depends on your isolation setting (mainly cgroup, or any node level resources). In general, we don't recommend folks use multiple agents on a node. It's possible to make it work by setting `cgroup_root` separately for MesosContainerizer. For DockerContainerizer, currently, we hard code `DOCKER_NAME_PREFIX`, making it not possible to use two agents on a node properly. - Jie On Wed, Mar 7, 2018 at 9:48 PM, Baskar Sikkayan wrote: > Hi, > > We are running 3 node mesos cluster and on each node running 1 mesos > master and 2 mesos slaves. Is this a good practice? > > Will it be a problem running 2 slaves as it might offer too much and will > get overloaded. > > > Thanks, > Baskar.S >
2 Mesos Slaves in same machine
Hi, We are running 3 node mesos cluster and on each node running 1 mesos master and 2 mesos slaves. Is this a good practice? Will it be a problem running 2 slaves as it might offer too much and will get overloaded. Thanks, Baskar.S
