[VOTE] Release Apache Mesos 1.0.0 (rc4)

[Vinod Kone]

Hi all,


Please vote on releasing the following candidate as Apache Mesos 1.0.0.

*The vote is open until Tue Jul 25 11:00:00 PDT 2016 and passes if a
majority of at least 3 +1 PMC votes are cast.*

1.0.0 includes the following:



  * Scheduler and Executor v1 HTTP APIs are now considered stable.





  * [MESOS-4791] - **Experimental** support for v1 Master and Agent APIs.
These

APIs let operators and services (monitoring, load balancers) send HTTP


requests to '/api/v1' endpoint on master or agent. See


`docs/operator-http-api.md` for details.





  * [MESOS-4828] - **Experimental** support for a new `disk/xfs' isolator


has been added to isolate disk resources more efficiently. Please refer
to

docs/mesos-containerizer.md for more details.





  * [MESOS-4355] - **Experimental** support for Docker volume plugin. We
added a

new isolator 'docker/volume' which allows users to use external volumes
in

Mesos containerizer. Currently, the isolator interacts with the Docker


volume plugins using a tool called 'dvdcli'. By speaking the Docker
volume

plugin API, most of the Docker volume plugins are supported.





  * [MESOS-4641] - **Experimental** A new network isolator, the


`network/cni` isolator, has been introduced in the
`MesosContainerizer`. The

`network/cni` isolator implements the Container Network Interface (CNI)


specification proposed by CoreOS.  With CNI the `network/cni` isolator
is

able to allocate a network namespace to Mesos containers and attach the


container to different types of IP networks by invoking network drivers


called CNI plugins.





  * [MESOS-2948, MESOS-5403] - The authorizer interface has been refactored
in

order to decouple the ACLs definition language from the interface.


It additionally includes the option of retrieving `ObjectApprover`. An


`ObjectApprover` can be used to synchronously check authorizations for
a

given object and is hence useful when authorizing a large number of
objects

and/or large objects (which need to be copied using request based


authorization). NOTE: This is a **breaking change** for authorizer
modules.




  * [MESOS-5405] - The `subject` and `object` fields in
authorization::Request

have been changed from required to optional. If either of these fields
is

not set, the request should only be authorized if any subject/object
should

be allowed.


NOTE: This is a semantic change for authorizer modules.





  * [MESOS-4931, MESOS-5709, MESOS-5704] - Authorization based HTTP
endpoint

filtering enables operators to restrict what part of the cluster state
a

user is authorized to see.


Consider for example the `/state` master endpoint: an operator can now


authorize users to only see a subset of the running frameworks, tasks,
or

Consider for example the `/state` master endpoint: an operator can now


authorize users to only see a subset of the running frameworks, tasks,
or

executors. The following endpoints support HTTP endpoint filtering:


'/state', '/state-summary', '/tasks', '/frameworks','/weights',


and '/roles'. Additonally the following v1 API calls support filtering:


'GET_ROLES','GET_WEIGHTS','GET_FRAMEWORKS', 'GET_STATE', and
'GET_TASKS'.




  * [MESOS-4909] - Tasks can now specify a kill policy. They are
best-effort,

because machine failures or forcible terminations may occur. Currently,
the

only available kill policy is how long to wait between graceful and
forcible

task kill. In the future, more policies may be available (e.g. hitting
an

HTTP endpoint, running a command, etc). Note that it is the executor's


responsibility to enforce kill policies. For executor-less
command-based

tasks, the kill is performed via sending a signal to the task process:


SIGTERM for the graceful kill and SIGKILL for the forcible kill. For
docker

executor-less tasks the grace period is passed to 'docker stop --time'.
This

feature supersedes the '--docker_stop_timeout', which is now
deprecated.




  * [MESOS-4908] - The task kill policy defined within 'TaskInfo' can now
be

overridden when the scheduler kills the task. This can be used by
schedulers

to forcefully kill a task which is already being killed, e.g. if
something

went wrong during a graceful kill and a forcible kill is desired. Note
that

it is the executor's responsibility to honor the
'Event.kill.kill_policy'

field and override the task's kill policy and kill policy from a
previous

kill task request. To use this feature, schedulers and executors must


support HTTP API; use the '--http_command_executor' agent flag to
ensure

the agent launches the HTTP API based command executor.





  * [MESOS-4949] - The executor shutdown grace period can now be configured
in

Re: [VOTE] Release Apache Mesos 1.0.0 (rc3)

[Vinod Kone]

Looks like we missed a cherry pick. I'm cancelling this vote and spinning
up rc4.

On Fri, Jul 22, 2016 at 2:24 PM, Vinod Kone  wrote:

> Hi all,
>
>
> Please vote on releasing the following candidate as Apache Mesos 1.0.0.
>
> *The vote is open until Tue Jul 25 11:00:00 PDT 2016 and passes if a
> majority of at least 3 +1 PMC votes are cast.*
>
> 1.0.0 includes the following:
>
>
> 
>
>   * Scheduler and Executor v1 HTTP APIs are now considered stable.
>
>
>
>
>
>   * [MESOS-4791] - **Experimental** support for v1 Master and Agent APIs.
> These
>
> APIs let operators and services (monitoring, load balancers) send
> HTTP
>
> requests to '/api/v1' endpoint on master or agent. See
>
>
> `docs/operator-http-api.md` for details.
>
>
>
>
>
>   * [MESOS-4828] - **Experimental** support for a new `disk/xfs' isolator
>
>
> has been added to isolate disk resources more efficiently. Please
> refer to
>
> docs/mesos-containerizer.md for more details.
>
>
>
>
>
>   * [MESOS-4355] - **Experimental** support for Docker volume plugin. We
> added a
>
> new isolator 'docker/volume' which allows users to use external
> volumes in
>
> Mesos containerizer. Currently, the isolator interacts with the
> Docker
>
> volume plugins using a tool called 'dvdcli'. By speaking the Docker
> volume
>
> plugin API, most of the Docker volume plugins are supported.
>
>
>
>
>
>   * [MESOS-4641] - **Experimental** A new network isolator, the
>
>
> `network/cni` isolator, has been introduced in the
> `MesosContainerizer`. The
>
> `network/cni` isolator implements the Container Network Interface
> (CNI)
>
> specification proposed by CoreOS.  With CNI the `network/cni` isolator
> is
>
> able to allocate a network namespace to Mesos containers and attach
> the
>
> container to different types of IP networks by invoking network
> drivers
>
> called CNI plugins.
>
>
>
>
>
>   * [MESOS-2948, MESOS-5403] - The authorizer interface has been
> refactored in
>
> order to decouple the ACLs definition language from the interface.
>
>
> It additionally includes the option of retrieving `ObjectApprover`.
> An
>
> `ObjectApprover` can be used to synchronously check authorizations for
> a
>
> given object and is hence useful when authorizing a large number of
> objects
>
> and/or large objects (which need to be copied using request based
>
>
> authorization). NOTE: This is a **breaking change** for authorizer
> modules.
>
>
>
>
>   * [MESOS-5405] - The `subject` and `object` fields in
> authorization::Request
>
> have been changed from required to optional. If either of these fields
> is
>
> not set, the request should only be authorized if any subject/object
> should
>
> be allowed.
>
>
> NOTE: This is a semantic change for authorizer modules.
>
>
>
>
>
>   * [MESOS-4931, MESOS-5709, MESOS-5704] - Authorization based HTTP
> endpoint
>
> filtering enables operators to restrict what part of the cluster state
> a
>
> user is authorized to see.
>
>
> Consider for example the `/state` master endpoint: an operator can
> now
>
> authorize users to only see a subset of the running frameworks, tasks,
> or
>
> Consider for example the `/state` master endpoint: an operator can
> now
>
> authorize users to only see a subset of the running frameworks, tasks,
> or
>
> executors. The following endpoints support HTTP endpoint filtering:
>
>
> '/state', '/state-summary', '/tasks', '/frameworks','/weights',
>
>
> and '/roles'. Additonally the following v1 API calls support
> filtering:
>
> 'GET_ROLES','GET_WEIGHTS','GET_FRAMEWORKS', 'GET_STATE', and
> 'GET_TASKS'.
>
>
>
>
>   * [MESOS-4909] - Tasks can now specify a kill policy. They are
> best-effort,
>
> because machine failures or forcible terminations may occur.
> Currently, the
>
> only available kill policy is how long to wait between graceful and
> forcible
>
> task kill. In the future, more policies may be available (e.g. hitting
> an
>
> HTTP endpoint, running a command, etc). Note that it is the
> executor's
>
> responsibility to enforce kill policies. For executor-less
> command-based
>
> tasks, the kill is performed via sending a signal to the task
> process:
>
> SIGTERM for the graceful kill and SIGKILL for the forcible kill. For
> docker
>
> executor-less tasks the grace period is passed to 'docker stop
> --time'. This
>
> feature supersedes the '--docker_stop_timeout', which is now
> deprecated.
>
>
>
>
>   * [MESOS-4908] - The task kill policy defined within 'TaskInfo' can now
> be
>
> overridden when the scheduler kills the task. This can be used by
> schedulers
>
> to forcefully kill a task which is already being killed, e.g. if
> something
>
> went wrong during a graceful kill and a forcible kill is desired. Note
> that
>

Re: What will happen in maintenance mode

[tommy xiao]

yes, in recently mesos deployment, if i ignore the hostname, just specified
IP, the mesos cluster sometime is not working. because the hostname is not
correct. so i also curious the machine definition:
"Each machine must have at least a hostname or IP included. The hostname is
not case-sensitive."

it should be defined must hostname and ip included.


2016-07-19 11:38 GMT+08:00 Qiang Chen :

> Thanks Joseph.
>
> I saw this from mesos [doc site](
> http://mesos.apache.org/documentation/latest/maintenance/):
>
> "Each machine must have at least a hostname or IP included. The hostname
> is not case-sensitive."
>
> From my test, the statement above is not correct, as if I only specific
> the hostname or IP, it will NOT take effect for the maintenance agents.
> but should specific both will OK.
>
> On 2016年07月19日 02:17, Joseph Wu wrote:
>
> [image: Boxbe]  This message is eligible
> for Automatic Cleanup! (jos...@mesosphere.io) Add cleanup rule
> 
> | More info
> 
>
>
> My guess is that your agents don't match the machines you specified.
> Note: The maintenance endpoints in Mesos allow you to specify maintenance
> against non-existent machines, because the operator may add agents on those
> machines in future.
>
> In Mesos' maintenance primitives, a "machine" is a hostname + IP.  (A
> physical/virtual machine can hold multiple agents.)  The response in
> /maintenance/status is in terms of machines, not agents.  If none of your
> frameworks support inverse offers, then you won't get any useful
> information from the /maintenance/status endpoint.
>
> You can figure out an agent's hostname/IP by hitting the /master/slaves
> endpoint:
>
> {
>   "slaves": [
> {
>   "pid":"slave(1)@127.0.0.1:5051",
>   "hostname":"foo-bar",
>   ...
>
> ^ The above translates to a machine = { "hostname": "foo-bar", "ip" : "
> 127.0.0.1" }
>
> On Mon, Jul 18, 2016 at 2:08 AM, Qiang Chen  wrote:
>
>> Hi all,
>>
>> I'm puzzled in using maintenance mode.
>>
>> I see this from mesos [doc site](
>> http://mesos.apache.org/documentation/latest/maintenance/):
>>
>> ```
>> When maintenance is triggered by the operator, all agents on the machine
>> are told to shutdown. These agents are removed from the master, which means
>> that a TASK_LOST status update will be sent for every task running on
>> each of those agents. The scheduler driver’s slaveLost callback will
>> also be invoked for each of the removed agents. Any agents on machines in
>> maintenance are also prevented from re-registering with the master in the
>> future (until maintenance is completed and the machine is brought back up).
>> ```
>> But I didn't find the agent machine shutdown or task failed when I test
>> the maintenance HTTP endpoints.
>>
>> If mesos agents are in that mode will move the running tasks to other
>> agents? namely, it will evacuate all the tasks in those agents? and the
>> shutdown?
>>
>> When I POST "/maintenance/schedule" and "/machine/down" and give a proper
>> maintain time window. I got the response that those specified agents are in
>> the "draining_machines" and "down_machines" list by GET
>> "/maintenance/status", but didn't shutdown and evacuate any tasks, why ?
>> does it make sense?
>>
>> Thanks.
>>
>> --
>> Best Regards,
>> Chen, Qiang
>>
>>
>
> --
> Best Regards,
> Chen, Qiang
>
>


-- 
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

Re: Marathon endpoint link and host networking

[tommy xiao]

cool tips.

2016-07-20 21:10 GMT+08:00 Matteo Piccinini <
matteo.piccin...@brightcomputing.com>:

> Hello everyone,
>
> Just fyi I solved the issue (Thanks to Tomek), I was submitting an
> incorrect JSON, missing ports and requirePorts parameters.
>
> Here the thread on the marathon list just in case someone will experience
> the same issue.
>
>
> https://groups.google.com/forum/?hl=en#!topic/marathon-framework/rtJIHUxxVTA
>
> Thanks.
>
> Best,
> Matteo
>



-- 
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

Re: Why not to use messaging queues

[tommy xiao]

i think the internal communication is HTTP over protobuf

2016-07-19 16:57 GMT+08:00 Sameer Rahmani :

> Hi folks.
>
> I was wondering why didn't mesos developers use a messaging queue instead
> of an HTTP API to serve frameworks ?
>
> Cheers
> Sameer
>



-- 
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

[VOTE] Release Apache Mesos 1.0.0 (rc3)

[Vinod Kone]

Hi all,


Please vote on releasing the following candidate as Apache Mesos 1.0.0.

*The vote is open until Tue Jul 25 11:00:00 PDT 2016 and passes if a
majority of at least 3 +1 PMC votes are cast.*

1.0.0 includes the following:



  * Scheduler and Executor v1 HTTP APIs are now considered stable.





  * [MESOS-4791] - **Experimental** support for v1 Master and Agent APIs.
These

APIs let operators and services (monitoring, load balancers) send HTTP


requests to '/api/v1' endpoint on master or agent. See


`docs/operator-http-api.md` for details.





  * [MESOS-4828] - **Experimental** support for a new `disk/xfs' isolator


has been added to isolate disk resources more efficiently. Please refer
to

docs/mesos-containerizer.md for more details.





  * [MESOS-4355] - **Experimental** support for Docker volume plugin. We
added a

new isolator 'docker/volume' which allows users to use external volumes
in

Mesos containerizer. Currently, the isolator interacts with the Docker


volume plugins using a tool called 'dvdcli'. By speaking the Docker
volume

plugin API, most of the Docker volume plugins are supported.





  * [MESOS-4641] - **Experimental** A new network isolator, the


`network/cni` isolator, has been introduced in the
`MesosContainerizer`. The

`network/cni` isolator implements the Container Network Interface (CNI)


specification proposed by CoreOS.  With CNI the `network/cni` isolator
is

able to allocate a network namespace to Mesos containers and attach the


container to different types of IP networks by invoking network drivers


called CNI plugins.





  * [MESOS-2948, MESOS-5403] - The authorizer interface has been refactored
in

order to decouple the ACLs definition language from the interface.


It additionally includes the option of retrieving `ObjectApprover`. An


`ObjectApprover` can be used to synchronously check authorizations for
a

given object and is hence useful when authorizing a large number of
objects

and/or large objects (which need to be copied using request based


authorization). NOTE: This is a **breaking change** for authorizer
modules.




  * [MESOS-5405] - The `subject` and `object` fields in
authorization::Request

have been changed from required to optional. If either of these fields
is

not set, the request should only be authorized if any subject/object
should

be allowed.


NOTE: This is a semantic change for authorizer modules.





  * [MESOS-4931, MESOS-5709, MESOS-5704] - Authorization based HTTP
endpoint

filtering enables operators to restrict what part of the cluster state
a

user is authorized to see.


Consider for example the `/state` master endpoint: an operator can now


authorize users to only see a subset of the running frameworks, tasks,
or

Consider for example the `/state` master endpoint: an operator can now


authorize users to only see a subset of the running frameworks, tasks,
or

executors. The following endpoints support HTTP endpoint filtering:


'/state', '/state-summary', '/tasks', '/frameworks','/weights',


and '/roles'. Additonally the following v1 API calls support filtering:


'GET_ROLES','GET_WEIGHTS','GET_FRAMEWORKS', 'GET_STATE', and
'GET_TASKS'.




  * [MESOS-4909] - Tasks can now specify a kill policy. They are
best-effort,

because machine failures or forcible terminations may occur. Currently,
the

only available kill policy is how long to wait between graceful and
forcible

task kill. In the future, more policies may be available (e.g. hitting
an

HTTP endpoint, running a command, etc). Note that it is the executor's


responsibility to enforce kill policies. For executor-less
command-based

tasks, the kill is performed via sending a signal to the task process:


SIGTERM for the graceful kill and SIGKILL for the forcible kill. For
docker

executor-less tasks the grace period is passed to 'docker stop --time'.
This

feature supersedes the '--docker_stop_timeout', which is now
deprecated.




  * [MESOS-4908] - The task kill policy defined within 'TaskInfo' can now
be

overridden when the scheduler kills the task. This can be used by
schedulers

to forcefully kill a task which is already being killed, e.g. if
something

went wrong during a graceful kill and a forcible kill is desired. Note
that

it is the executor's responsibility to honor the
'Event.kill.kill_policy'

field and override the task's kill policy and kill policy from a
previous

kill task request. To use this feature, schedulers and executors must


support HTTP API; use the '--http_command_executor' agent flag to
ensure

the agent launches the HTTP API based command executor.





  * [MESOS-4949] - The executor shutdown grace period can now be configured
in

Re: Releasing a Scala API for Mesos

[Attila Szarvas]

Thanks for the suggestion, I've sent the pull request.


On 2016-07-22 12:04, haosdent wrote:
Thanks a lot for your sharing. May you mind open a pull request on 
github to add your project to 
https://github.com/apache/mesos/blob/master/docs/api-client-libraries.md ?


On Fri, Jul 22, 2016 at 5:51 PM, Szarvas, Attila (Nokia - HU/Budapest) 
> wrote:


Hello,

I am working for Nokia Bell Labs and we have been experimenting
with Mesos, and we had really good impressions while interacting
with it from a Scala project.

We have separated the parts responsible for interfacing with Mesos
from Scala and now releasing it as a library.
http://github.com/nokia/mesos-scala-api

It builds on top of the Java API and provides an idiomatic way of
interacting with Mesos, notably providing reactive event streams
preferable to the callback based Java API.

Beyond the scheduler and scheduler driver API, it also contains an
easy to use framework implementation. In as much as 30 lines of
code one can connect to a Mesos master, launch a task, observe
events pertaining to it, and disconnect after successful execution.

We'd love to hear any feedback you might have about it, whether it
may be of use to you, or if you could imagine it as part of an
official distribution of Mesos.

We are happy to answer your questions.
Attila Szarvas - attila.szar...@nokia-bell-labs.com

Daniel Urban - daniel.ur...@nokia-bell-labs.com


Best regards,

Attila Szarvas
Research Engineer
Nokia Bell Labs, Budapest




--
Best Regards,
Haosdent Huang


--
Br,
Attila Szarvas

Re: Releasing a Scala API for Mesos

[haosdent]

Thanks a lot for your sharing. May you mind open a pull request on github
to add your project to
https://github.com/apache/mesos/blob/master/docs/api-client-libraries.md ?

On Fri, Jul 22, 2016 at 5:51 PM, Szarvas, Attila (Nokia - HU/Budapest) <
attila.szar...@nokia-bell-labs.com> wrote:

> Hello,
>
> I am working for Nokia Bell Labs and we have been experimenting with
> Mesos, and we had really good impressions while interacting with it from a
> Scala project.
>
> We have separated the parts responsible for interfacing with Mesos from
> Scala and now releasing it as a library.
> http://github.com/nokia/mesos-scala-api
>
> It builds on top of the Java API and provides an idiomatic way of
> interacting with Mesos, notably providing reactive event streams preferable
> to the callback based Java API.
>
> Beyond the scheduler and scheduler driver API, it also contains an easy to
> use framework implementation. In as much as 30 lines of code one can
> connect to a Mesos master, launch a task, observe events pertaining to it,
> and disconnect after successful execution.
>
> We'd love to hear any feedback you might have about it, whether it may be
> of use to you, or if you could imagine it as part of an official
> distribution of Mesos.
>
> We are happy to answer your questions.
> Attila Szarvas - attila.szar...@nokia-bell-labs.com
> Daniel Urban - daniel.ur...@nokia-bell-labs.com
>
> Best regards,
>
> Attila Szarvas
> Research Engineer
> Nokia Bell Labs, Budapest




-- 
Best Regards,
Haosdent Huang

Releasing a Scala API for Mesos

[Szarvas, Attila (Nokia - HU/Budapest)]

Hello,

I am working for Nokia Bell Labs and we have been experimenting with Mesos, and 
we had really good impressions while interacting with it from a Scala project.

We have separated the parts responsible for interfacing with Mesos from Scala 
and now releasing it as a library.
http://github.com/nokia/mesos-scala-api

It builds on top of the Java API and provides an idiomatic way of interacting 
with Mesos, notably providing reactive event streams preferable to the callback 
based Java API.

Beyond the scheduler and scheduler driver API, it also contains an easy to use 
framework implementation. In as much as 30 lines of code one can connect to a 
Mesos master, launch a task, observe events pertaining to it, and disconnect 
after successful execution.

We'd love to hear any feedback you might have about it, whether it may be of 
use to you, or if you could imagine it as part of an official distribution of 
Mesos.

We are happy to answer your questions.
Attila Szarvas - attila.szar...@nokia-bell-labs.com
Daniel Urban - daniel.ur...@nokia-bell-labs.com

Best regards,

Attila Szarvas
Research Engineer
Nokia Bell Labs, Budapest