Re: Allowing both CommandInfo and ExecutorInfo on TaskInfo

[Zameer Manji]

It isn't issue if it a broadening of possibilities.

I would like to point out that, if we do want to have a better contract in
the future (ie a set of CommandInfos), we are setting up ourselves for a
API change in the future.

If we clearly document that the CommandInfo is just passed down to the
executor by the master and agent, I don't see any harm.


On Fri, Nov 4, 2016 at 6:53 AM, Joris Van Remoortere 
wrote:

> @zameer
>
> I think your example makes a lot of sense. I didn't interpret the proposal
> as one that would prevent your case.
>
> The way I read the proposal is that we want to allow setting both, not
> require it. If I misunderstood then please ignore my comments.
>
> When I first heard about this proposal it seemed like a nice way for
> frameworks and executors to start defining a more structured contract _if
> they wanted to_, while still allowing others to keep passing their (from
> Mesos's view) unstructured data as long as both sides agree on the
> serialization / de-serialization.
>
> Does this still seem like an issue if it's not a requirement but just a
> broadening of possibilities?
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Wed, Nov 2, 2016 at 1:49 PM, Zameer Manji  wrote:
>
> > Joris,
> >
> > You make a good point. However, I'm not convinced that `CommandInfo`
> should
> > be the well defined construct that people use. Can you please describe
> > different custom executors, and the overlap between them and how
> > CommandInfo will reduce that overlap? I'm having a hard time seeing where
> > CommandInfo will solve all of their cases.
> >
> > Consider the cause of Thermos (Aurora's Executor), it could never use a
> > `CommandInfo` struct because it executes a processes graph instead of a
> > single command.
> >
> > If the project wants to go down this path, I think generalizing
> > `CommandInfo` that could capture more cases (ie multiple commands or a
> > graph of commands) would be a better first step.
> >
> > What do you think?
> >
> > On Wed, Oct 26, 2016 at 10:38 AM, Joris Van Remoortere <
> > jo...@mesosphere.io>
> > wrote:
> >
> > > I do think it would be valuable to have a more well defined contract
> > > between frameworks and custom executors.
> > >
> > > As Zameer pointed out a specific framework and accompanying custom
> > executor
> > > can decide to do that in the data bytes; however, if we started
> building
> > > out a few different flavors of executors then it would be great for
> there
> > > to be standard way to pass command information to them.
> > >
> > > The current model works well in a 1-1 mapping between framework and
> > > executor binaries. In a world where that is 1-N it means all N
> executors
> > > have to use the same method of passing the command.
> > >
> > > —
> > > *Joris Van Remoortere*
> > > Mesosphere
> > >
> > > On Mon, Oct 17, 2016 at 4:25 PM, Zameer Manji 
> wrote:
> > >
> > > > I'm not convinced this is a valid use case.
> > > >
> > > > Mesos is supposed to be a generic kernel for launching "tasks",
> > whatever
> > > > they might be.
> > > >
> > > > In some cases it is useful to launch an executable, in other cases it
> > > might
> > > > be useful to launch a series of executables, and in some other cases
> it
> > > > might be useful to spawn a thread to do some work. Whatever that
> might
> > > be,
> > > > it doesn't matter to Mesos and the executor and framework are free to
> > > > establish a contract in `ExecutorInfo.data`, completely independent
> of
> > > the
> > > > Mesos API.
> > > >
> > > > I think formalizing this contract between executors and frameworks
> via
> > > > CommandInfo is going to introduce more problems than what they solve.
> > If
> > > > the CommandInfo struct is useful, frameworks and executors can just
> > stuff
> > > > that into ExecutorInfo.data, however it's not something that they
> need
> > to
> > > > adhere too.
> > > >
> > > > What's the underlying motivation for this?
> > > >
> > > >
> > > >
> > > > On Thu, Oct 13, 2016 at 10:40 AM, haosdent 
> wrote:
> > > >
> > > > > For command task, if its `ExecutorInfo` would set with
> > > `CommandExecutor`
> > > > as
> > > > > well?
> > > > >
> > > > > Some tickets may relate to this.
> > > > >
> > > > > [1]: https://issues.apache.org/jira/browse/MESOS-2330
> > > > > [2]: https://issues.apache.org/jira/browse/MESOS-527
> > > > > [3]: https://issues.apache.org/jira/browse/MESOS-5198
> > > > >
> > > > > On Fri, Oct 14, 2016 at 1:00 AM, Vinod Kone 
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We are contemplating whether to allow both CommandInfo and
> > > ExecutorInfo
> > > > > on
> > > > > > TaskInfo (MESOS-6294  > > jira/browse/MESOS-6294
> > > > > >).
> > > > > > Currently we only allow one or the other. The motivation is to
> > allow
> > > > > custom
> > > > > > executors a more structured way to pass 

Re: Allowing both CommandInfo and ExecutorInfo on TaskInfo

[Joris Van Remoortere]

@zameer

I think your example makes a lot of sense. I didn't interpret the proposal
as one that would prevent your case.

The way I read the proposal is that we want to allow setting both, not
require it. If I misunderstood then please ignore my comments.

When I first heard about this proposal it seemed like a nice way for
frameworks and executors to start defining a more structured contract _if
they wanted to_, while still allowing others to keep passing their (from
Mesos's view) unstructured data as long as both sides agree on the
serialization / de-serialization.

Does this still seem like an issue if it's not a requirement but just a
broadening of possibilities?

—
*Joris Van Remoortere*
Mesosphere

On Wed, Nov 2, 2016 at 1:49 PM, Zameer Manji  wrote:

> Joris,
>
> You make a good point. However, I'm not convinced that `CommandInfo` should
> be the well defined construct that people use. Can you please describe
> different custom executors, and the overlap between them and how
> CommandInfo will reduce that overlap? I'm having a hard time seeing where
> CommandInfo will solve all of their cases.
>
> Consider the cause of Thermos (Aurora's Executor), it could never use a
> `CommandInfo` struct because it executes a processes graph instead of a
> single command.
>
> If the project wants to go down this path, I think generalizing
> `CommandInfo` that could capture more cases (ie multiple commands or a
> graph of commands) would be a better first step.
>
> What do you think?
>
> On Wed, Oct 26, 2016 at 10:38 AM, Joris Van Remoortere <
> jo...@mesosphere.io>
> wrote:
>
> > I do think it would be valuable to have a more well defined contract
> > between frameworks and custom executors.
> >
> > As Zameer pointed out a specific framework and accompanying custom
> executor
> > can decide to do that in the data bytes; however, if we started building
> > out a few different flavors of executors then it would be great for there
> > to be standard way to pass command information to them.
> >
> > The current model works well in a 1-1 mapping between framework and
> > executor binaries. In a world where that is 1-N it means all N executors
> > have to use the same method of passing the command.
> >
> > —
> > *Joris Van Remoortere*
> > Mesosphere
> >
> > On Mon, Oct 17, 2016 at 4:25 PM, Zameer Manji  wrote:
> >
> > > I'm not convinced this is a valid use case.
> > >
> > > Mesos is supposed to be a generic kernel for launching "tasks",
> whatever
> > > they might be.
> > >
> > > In some cases it is useful to launch an executable, in other cases it
> > might
> > > be useful to launch a series of executables, and in some other cases it
> > > might be useful to spawn a thread to do some work. Whatever that might
> > be,
> > > it doesn't matter to Mesos and the executor and framework are free to
> > > establish a contract in `ExecutorInfo.data`, completely independent of
> > the
> > > Mesos API.
> > >
> > > I think formalizing this contract between executors and frameworks via
> > > CommandInfo is going to introduce more problems than what they solve.
> If
> > > the CommandInfo struct is useful, frameworks and executors can just
> stuff
> > > that into ExecutorInfo.data, however it's not something that they need
> to
> > > adhere too.
> > >
> > > What's the underlying motivation for this?
> > >
> > >
> > >
> > > On Thu, Oct 13, 2016 at 10:40 AM, haosdent  wrote:
> > >
> > > > For command task, if its `ExecutorInfo` would set with
> > `CommandExecutor`
> > > as
> > > > well?
> > > >
> > > > Some tickets may relate to this.
> > > >
> > > > [1]: https://issues.apache.org/jira/browse/MESOS-2330
> > > > [2]: https://issues.apache.org/jira/browse/MESOS-527
> > > > [3]: https://issues.apache.org/jira/browse/MESOS-5198
> > > >
> > > > On Fri, Oct 14, 2016 at 1:00 AM, Vinod Kone 
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We are contemplating whether to allow both CommandInfo and
> > ExecutorInfo
> > > > on
> > > > > TaskInfo (MESOS-6294  > jira/browse/MESOS-6294
> > > > >).
> > > > > Currently we only allow one or the other. The motivation is to
> allow
> > > > custom
> > > > > executors a more structured way to pass information (e.g, command)
> > > about
> > > > > Task. Right now custom executors have to get this data via
> > > > `TaskInfo.bytes`
> > > > > which is not ideal.
> > > > >
> > > > > Are there any custom executors out there that crash if they get
> Tasks
> > > > with
> > > > > CommandInfo set?
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Vinod
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards,
> > > > Haosdent Huang
> > > >
> > > > --
> > > > Zameer Manji
> > > >
> > >
> >
> > --
> > Zameer Manji
> >
>

Re: Allowing both CommandInfo and ExecutorInfo on TaskInfo

[David Greenberg]

I have worked with executors that perform both conditional & unconditional
execution of a process graph, configurable by the framework. I think that
it would be hard to standardize.

On Wed, Nov 2, 2016 at 2:49 PM Zameer Manji  wrote:

> Joris,
>
> You make a good point. However, I'm not convinced that `CommandInfo` should
> be the well defined construct that people use. Can you please describe
> different custom executors, and the overlap between them and how
> CommandInfo will reduce that overlap? I'm having a hard time seeing where
> CommandInfo will solve all of their cases.
>
> Consider the cause of Thermos (Aurora's Executor), it could never use a
> `CommandInfo` struct because it executes a processes graph instead of a
> single command.
>
> If the project wants to go down this path, I think generalizing
> `CommandInfo` that could capture more cases (ie multiple commands or a
> graph of commands) would be a better first step.
>
> What do you think?
>
> On Wed, Oct 26, 2016 at 10:38 AM, Joris Van Remoortere <
> jo...@mesosphere.io>
> wrote:
>
> > I do think it would be valuable to have a more well defined contract
> > between frameworks and custom executors.
> >
> > As Zameer pointed out a specific framework and accompanying custom
> executor
> > can decide to do that in the data bytes; however, if we started building
> > out a few different flavors of executors then it would be great for there
> > to be standard way to pass command information to them.
> >
> > The current model works well in a 1-1 mapping between framework and
> > executor binaries. In a world where that is 1-N it means all N executors
> > have to use the same method of passing the command.
> >
> > —
> > *Joris Van Remoortere*
> > Mesosphere
> >
> > On Mon, Oct 17, 2016 at 4:25 PM, Zameer Manji  wrote:
> >
> > > I'm not convinced this is a valid use case.
> > >
> > > Mesos is supposed to be a generic kernel for launching "tasks",
> whatever
> > > they might be.
> > >
> > > In some cases it is useful to launch an executable, in other cases it
> > might
> > > be useful to launch a series of executables, and in some other cases it
> > > might be useful to spawn a thread to do some work. Whatever that might
> > be,
> > > it doesn't matter to Mesos and the executor and framework are free to
> > > establish a contract in `ExecutorInfo.data`, completely independent of
> > the
> > > Mesos API.
> > >
> > > I think formalizing this contract between executors and frameworks via
> > > CommandInfo is going to introduce more problems than what they solve.
> If
> > > the CommandInfo struct is useful, frameworks and executors can just
> stuff
> > > that into ExecutorInfo.data, however it's not something that they need
> to
> > > adhere too.
> > >
> > > What's the underlying motivation for this?
> > >
> > >
> > >
> > > On Thu, Oct 13, 2016 at 10:40 AM, haosdent  wrote:
> > >
> > > > For command task, if its `ExecutorInfo` would set with
> > `CommandExecutor`
> > > as
> > > > well?
> > > >
> > > > Some tickets may relate to this.
> > > >
> > > > [1]: https://issues.apache.org/jira/browse/MESOS-2330
> > > > [2]: https://issues.apache.org/jira/browse/MESOS-527
> > > > [3]: https://issues.apache.org/jira/browse/MESOS-5198
> > > >
> > > > On Fri, Oct 14, 2016 at 1:00 AM, Vinod Kone 
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We are contemplating whether to allow both CommandInfo and
> > ExecutorInfo
> > > > on
> > > > > TaskInfo (MESOS-6294  > jira/browse/MESOS-6294
> > > > >).
> > > > > Currently we only allow one or the other. The motivation is to
> allow
> > > > custom
> > > > > executors a more structured way to pass information (e.g, command)
> > > about
> > > > > Task. Right now custom executors have to get this data via
> > > > `TaskInfo.bytes`
> > > > > which is not ideal.
> > > > >
> > > > > Are there any custom executors out there that crash if they get
> Tasks
> > > > with
> > > > > CommandInfo set?
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Vinod
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards,
> > > > Haosdent Huang
> > > >
> > > > --
> > > > Zameer Manji
> > > >
> > >
> >
> > --
> > Zameer Manji
> >
>

Re: Allowing both CommandInfo and ExecutorInfo on TaskInfo

[Zameer Manji]

Joris,

You make a good point. However, I'm not convinced that `CommandInfo` should
be the well defined construct that people use. Can you please describe
different custom executors, and the overlap between them and how
CommandInfo will reduce that overlap? I'm having a hard time seeing where
CommandInfo will solve all of their cases.

Consider the cause of Thermos (Aurora's Executor), it could never use a
`CommandInfo` struct because it executes a processes graph instead of a
single command.

If the project wants to go down this path, I think generalizing
`CommandInfo` that could capture more cases (ie multiple commands or a
graph of commands) would be a better first step.

What do you think?

On Wed, Oct 26, 2016 at 10:38 AM, Joris Van Remoortere 
wrote:

> I do think it would be valuable to have a more well defined contract
> between frameworks and custom executors.
>
> As Zameer pointed out a specific framework and accompanying custom executor
> can decide to do that in the data bytes; however, if we started building
> out a few different flavors of executors then it would be great for there
> to be standard way to pass command information to them.
>
> The current model works well in a 1-1 mapping between framework and
> executor binaries. In a world where that is 1-N it means all N executors
> have to use the same method of passing the command.
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Mon, Oct 17, 2016 at 4:25 PM, Zameer Manji  wrote:
>
> > I'm not convinced this is a valid use case.
> >
> > Mesos is supposed to be a generic kernel for launching "tasks", whatever
> > they might be.
> >
> > In some cases it is useful to launch an executable, in other cases it
> might
> > be useful to launch a series of executables, and in some other cases it
> > might be useful to spawn a thread to do some work. Whatever that might
> be,
> > it doesn't matter to Mesos and the executor and framework are free to
> > establish a contract in `ExecutorInfo.data`, completely independent of
> the
> > Mesos API.
> >
> > I think formalizing this contract between executors and frameworks via
> > CommandInfo is going to introduce more problems than what they solve. If
> > the CommandInfo struct is useful, frameworks and executors can just stuff
> > that into ExecutorInfo.data, however it's not something that they need to
> > adhere too.
> >
> > What's the underlying motivation for this?
> >
> >
> >
> > On Thu, Oct 13, 2016 at 10:40 AM, haosdent  wrote:
> >
> > > For command task, if its `ExecutorInfo` would set with
> `CommandExecutor`
> > as
> > > well?
> > >
> > > Some tickets may relate to this.
> > >
> > > [1]: https://issues.apache.org/jira/browse/MESOS-2330
> > > [2]: https://issues.apache.org/jira/browse/MESOS-527
> > > [3]: https://issues.apache.org/jira/browse/MESOS-5198
> > >
> > > On Fri, Oct 14, 2016 at 1:00 AM, Vinod Kone 
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > We are contemplating whether to allow both CommandInfo and
> ExecutorInfo
> > > on
> > > > TaskInfo (MESOS-6294  jira/browse/MESOS-6294
> > > >).
> > > > Currently we only allow one or the other. The motivation is to allow
> > > custom
> > > > executors a more structured way to pass information (e.g, command)
> > about
> > > > Task. Right now custom executors have to get this data via
> > > `TaskInfo.bytes`
> > > > which is not ideal.
> > > >
> > > > Are there any custom executors out there that crash if they get Tasks
> > > with
> > > > CommandInfo set?
> > > >
> > > > Thoughts?
> > > >
> > > > Vinod
> > > >
> > >
> > >
> > >
> > > --
> > > Best Regards,
> > > Haosdent Huang
> > >
> > > --
> > > Zameer Manji
> > >
> >
>
> --
> Zameer Manji
>

Re: Allowing both CommandInfo and ExecutorInfo on TaskInfo

[Joris Van Remoortere]

I do think it would be valuable to have a more well defined contract
between frameworks and custom executors.

As Zameer pointed out a specific framework and accompanying custom executor
can decide to do that in the data bytes; however, if we started building
out a few different flavors of executors then it would be great for there
to be standard way to pass command information to them.

The current model works well in a 1-1 mapping between framework and
executor binaries. In a world where that is 1-N it means all N executors
have to use the same method of passing the command.

—
*Joris Van Remoortere*
Mesosphere

On Mon, Oct 17, 2016 at 4:25 PM, Zameer Manji  wrote:

> I'm not convinced this is a valid use case.
>
> Mesos is supposed to be a generic kernel for launching "tasks", whatever
> they might be.
>
> In some cases it is useful to launch an executable, in other cases it might
> be useful to launch a series of executables, and in some other cases it
> might be useful to spawn a thread to do some work. Whatever that might be,
> it doesn't matter to Mesos and the executor and framework are free to
> establish a contract in `ExecutorInfo.data`, completely independent of the
> Mesos API.
>
> I think formalizing this contract between executors and frameworks via
> CommandInfo is going to introduce more problems than what they solve. If
> the CommandInfo struct is useful, frameworks and executors can just stuff
> that into ExecutorInfo.data, however it's not something that they need to
> adhere too.
>
> What's the underlying motivation for this?
>
>
>
> On Thu, Oct 13, 2016 at 10:40 AM, haosdent  wrote:
>
> > For command task, if its `ExecutorInfo` would set with `CommandExecutor`
> as
> > well?
> >
> > Some tickets may relate to this.
> >
> > [1]: https://issues.apache.org/jira/browse/MESOS-2330
> > [2]: https://issues.apache.org/jira/browse/MESOS-527
> > [3]: https://issues.apache.org/jira/browse/MESOS-5198
> >
> > On Fri, Oct 14, 2016 at 1:00 AM, Vinod Kone 
> wrote:
> >
> > > Hi,
> > >
> > > We are contemplating whether to allow both CommandInfo and ExecutorInfo
> > on
> > > TaskInfo (MESOS-6294  > >).
> > > Currently we only allow one or the other. The motivation is to allow
> > custom
> > > executors a more structured way to pass information (e.g, command)
> about
> > > Task. Right now custom executors have to get this data via
> > `TaskInfo.bytes`
> > > which is not ideal.
> > >
> > > Are there any custom executors out there that crash if they get Tasks
> > with
> > > CommandInfo set?
> > >
> > > Thoughts?
> > >
> > > Vinod
> > >
> >
> >
> >
> > --
> > Best Regards,
> > Haosdent Huang
> >
> > --
> > Zameer Manji
> >
>

Re: Allowing both CommandInfo and ExecutorInfo on TaskInfo

[Zameer Manji]

I'm not convinced this is a valid use case.

Mesos is supposed to be a generic kernel for launching "tasks", whatever
they might be.

In some cases it is useful to launch an executable, in other cases it might
be useful to launch a series of executables, and in some other cases it
might be useful to spawn a thread to do some work. Whatever that might be,
it doesn't matter to Mesos and the executor and framework are free to
establish a contract in `ExecutorInfo.data`, completely independent of the
Mesos API.

I think formalizing this contract between executors and frameworks via
CommandInfo is going to introduce more problems than what they solve. If
the CommandInfo struct is useful, frameworks and executors can just stuff
that into ExecutorInfo.data, however it's not something that they need to
adhere too.

What's the underlying motivation for this?



On Thu, Oct 13, 2016 at 10:40 AM, haosdent  wrote:

> For command task, if its `ExecutorInfo` would set with `CommandExecutor` as
> well?
>
> Some tickets may relate to this.
>
> [1]: https://issues.apache.org/jira/browse/MESOS-2330
> [2]: https://issues.apache.org/jira/browse/MESOS-527
> [3]: https://issues.apache.org/jira/browse/MESOS-5198
>
> On Fri, Oct 14, 2016 at 1:00 AM, Vinod Kone  wrote:
>
> > Hi,
> >
> > We are contemplating whether to allow both CommandInfo and ExecutorInfo
> on
> > TaskInfo (MESOS-6294  >).
> > Currently we only allow one or the other. The motivation is to allow
> custom
> > executors a more structured way to pass information (e.g, command) about
> > Task. Right now custom executors have to get this data via
> `TaskInfo.bytes`
> > which is not ideal.
> >
> > Are there any custom executors out there that crash if they get Tasks
> with
> > CommandInfo set?
> >
> > Thoughts?
> >
> > Vinod
> >
>
>
>
> --
> Best Regards,
> Haosdent Huang
>
> --
> Zameer Manji
>

Re: Allowing both CommandInfo and ExecutorInfo on TaskInfo

[haosdent]

For command task, if its `ExecutorInfo` would set with `CommandExecutor` as
well?

Some tickets may relate to this.

[1]: https://issues.apache.org/jira/browse/MESOS-2330
[2]: https://issues.apache.org/jira/browse/MESOS-527
[3]: https://issues.apache.org/jira/browse/MESOS-5198

On Fri, Oct 14, 2016 at 1:00 AM, Vinod Kone  wrote:

> Hi,
>
> We are contemplating whether to allow both CommandInfo and ExecutorInfo on
> TaskInfo (MESOS-6294 ).
> Currently we only allow one or the other. The motivation is to allow custom
> executors a more structured way to pass information (e.g, command) about
> Task. Right now custom executors have to get this data via `TaskInfo.bytes`
> which is not ideal.
>
> Are there any custom executors out there that crash if they get Tasks with
> CommandInfo set?
>
> Thoughts?
>
> Vinod
>



-- 
Best Regards,
Haosdent Huang

Allowing both CommandInfo and ExecutorInfo on TaskInfo

[Vinod Kone]

Hi,

We are contemplating whether to allow both CommandInfo and ExecutorInfo on
TaskInfo (MESOS-6294 ).
Currently we only allow one or the other. The motivation is to allow custom
executors a more structured way to pass information (e.g, command) about
Task. Right now custom executors have to get this data via `TaskInfo.bytes`
which is not ideal.

Are there any custom executors out there that crash if they get Tasks with
CommandInfo set?

Thoughts?

Vinod