Re: 答复: Question about "Framework directly access Meso agent"

[Klaus Ma]

Suteng,

Regarding the storage/data part, agree with Joris to use distributed
FS/Cache for data.
Regarding resource bookkeeping, Master also need to check whether framework
used more resources against its offers. If framework access Mesos agent to
launch tasks, it can not sync up Master/Agent timely.



On Thu, Feb 18, 2016 at 10:22 PM Joris Van Remoortere <jo...@mesosphere.io>
wrote:

> If the main reason for contemplating this design is the size of the task
> payload, have you considered a content addressable storage design?
> For example: why can the task not be launched with URIs that allow the
> agent to download the payload before launching the task? This fans out the
> network load evenly among agents.
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Wed, Feb 17, 2016 at 9:35 PM, Suteng <sut...@huawei.com> wrote:
>
> > Alex,
> > We don't have test the performance of mesos. But we have develop a
> > framework in house, which is like a simplified mesos, use to schedule a
> > large number fine grain computation tasks.  We find that master will be a
> > bottleneck. One reason is our task contain several KB data, and task
> number
> > is quite huge.
> > If we use mesos to replace it, maybe master still be a bottleneck.
> >
> > Master still do the resource bookkeeping, we can decompose launch task to
> > two steps, firstly scheduler tell master which offer he wants, then
> master
> > tell scheduler the address of agent. Secondly, scheduler can directly
> > launch task to the agent, and also can directly send message to agent.
> > Maybe I can do some test about the mesos master launch task throughput,
> > with different number task data.
> >
> >
> > -----邮件原件-
> > 发件人: C Rukletsov [mailto:a...@mesosphere.com]
> > 发送时间: 2016年2月17日 18:04
> > 收件人: dev
> > 主题: Re: Question about "Framework directly access Meso agent"
> >
> > Suteng—
> >
> > such optimization makes sense in certain cases (e.g. sending a framework
> > message), but it can be rather tricky in general, because the master has
> to
> > maintain bookkeeping. Moreover, with the upcoming HTTP API it becomes
> > harder for a framework to determine where to send messages to reach a
> > specific agent.
> >
> > Have you done any performance tests and seen master becoming a
> bottleneck?
> >
> > On Wed, Feb 17, 2016 at 5:14 AM, Suteng <sut...@huawei.com> wrote:
> >
> > > Hi,
> > >
> > >
> > >
> > > Currently, Mesos framework’s task related operations lauchTask,
> > > updateStatus and executorSendMessage etc., and resource related
> > > operations resourceOffer etc., all operations are pass through Mesos
> > Master.
> > >
> > > When the cluster and task number become huge, or with optimistic
> > > resource offer, multi-framework concurrently launchTask, maybe Mesos
> > > Master will be a bottleneck.
> > >
> > > Is possible for framework scheduler directly access Mesos agent,
> > > launchTask, updateStatus and SendMessage2Executore to Mesos Agent
> > > directly, bypass the Master?
> > >
> > > Will invoke big conflict with current mechanism?
> > >
> > >
> > >
> > > Looking forward to your comments and opinions.
> > >
> > >
> > >
> > > Best Regards,
> > >
> > > Teng
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Su Teng  00241668
> > >
> > >
> > >
> > > Distributed and Parallel Software Lab
> > >
> > > Huawei Technologies Co., Ltd.
> > >
> > > Email:sut...@huawei.com
> > >
> > >
> > >
> > >
> > >
> >
>
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Advisory Software Engineer
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: 答复: Question about "Framework directly access Meso agent"

[Joris Van Remoortere]

If the main reason for contemplating this design is the size of the task
payload, have you considered a content addressable storage design?
For example: why can the task not be launched with URIs that allow the
agent to download the payload before launching the task? This fans out the
network load evenly among agents.

—
*Joris Van Remoortere*
Mesosphere

On Wed, Feb 17, 2016 at 9:35 PM, Suteng <sut...@huawei.com> wrote:

> Alex,
> We don't have test the performance of mesos. But we have develop a
> framework in house, which is like a simplified mesos, use to schedule a
> large number fine grain computation tasks.  We find that master will be a
> bottleneck. One reason is our task contain several KB data, and task number
> is quite huge.
> If we use mesos to replace it, maybe master still be a bottleneck.
>
> Master still do the resource bookkeeping, we can decompose launch task to
> two steps, firstly scheduler tell master which offer he wants, then master
> tell scheduler the address of agent. Secondly, scheduler can directly
> launch task to the agent, and also can directly send message to agent.
> Maybe I can do some test about the mesos master launch task throughput,
> with different number task data.
>
>
> -邮件原件-
> 发件人: C Rukletsov [mailto:a...@mesosphere.com]
> 发送时间: 2016年2月17日 18:04
> 收件人: dev
> 主题: Re: Question about "Framework directly access Meso agent"
>
> Suteng—
>
> such optimization makes sense in certain cases (e.g. sending a framework
> message), but it can be rather tricky in general, because the master has to
> maintain bookkeeping. Moreover, with the upcoming HTTP API it becomes
> harder for a framework to determine where to send messages to reach a
> specific agent.
>
> Have you done any performance tests and seen master becoming a bottleneck?
>
> On Wed, Feb 17, 2016 at 5:14 AM, Suteng <sut...@huawei.com> wrote:
>
> > Hi,
> >
> >
> >
> > Currently, Mesos framework’s task related operations lauchTask,
> > updateStatus and executorSendMessage etc., and resource related
> > operations resourceOffer etc., all operations are pass through Mesos
> Master.
> >
> > When the cluster and task number become huge, or with optimistic
> > resource offer, multi-framework concurrently launchTask, maybe Mesos
> > Master will be a bottleneck.
> >
> > Is possible for framework scheduler directly access Mesos agent,
> > launchTask, updateStatus and SendMessage2Executore to Mesos Agent
> > directly, bypass the Master?
> >
> > Will invoke big conflict with current mechanism?
> >
> >
> >
> > Looking forward to your comments and opinions.
> >
> >
> >
> > Best Regards,
> >
> > Teng
> >
> >
> >
> >
> >
> >
> >
> > Su Teng  00241668
> >
> >
> >
> > Distributed and Parallel Software Lab
> >
> > Huawei Technologies Co., Ltd.
> >
> > Email:sut...@huawei.com
> >
> >
> >
> >
> >
>

答复: Question about "Framework directly access Meso agent"

[Suteng]

Alex,
We don't have test the performance of mesos. But we have develop a framework in 
house, which is like a simplified mesos, use to schedule a large number fine 
grain computation tasks.  We find that master will be a bottleneck. One reason 
is our task contain several KB data, and task number is quite huge. 
If we use mesos to replace it, maybe master still be a bottleneck.

Master still do the resource bookkeeping, we can decompose launch task to two 
steps, firstly scheduler tell master which offer he wants, then master tell 
scheduler the address of agent. Secondly, scheduler can directly launch task to 
the agent, and also can directly send message to agent.
Maybe I can do some test about the mesos master launch task throughput, with 
different number task data.


-邮件原件-
发件人: C Rukletsov [mailto:a...@mesosphere.com] 
发送时间: 2016年2月17日 18:04
收件人: dev
主题: Re: Question about "Framework directly access Meso agent"

Suteng—

such optimization makes sense in certain cases (e.g. sending a framework 
message), but it can be rather tricky in general, because the master has to 
maintain bookkeeping. Moreover, with the upcoming HTTP API it becomes harder 
for a framework to determine where to send messages to reach a specific agent.

Have you done any performance tests and seen master becoming a bottleneck?

On Wed, Feb 17, 2016 at 5:14 AM, Suteng <sut...@huawei.com> wrote:

> Hi,
>
>
>
> Currently, Mesos framework’s task related operations lauchTask, 
> updateStatus and executorSendMessage etc., and resource related 
> operations resourceOffer etc., all operations are pass through Mesos Master.
>
> When the cluster and task number become huge, or with optimistic 
> resource offer, multi-framework concurrently launchTask, maybe Mesos 
> Master will be a bottleneck.
>
> Is possible for framework scheduler directly access Mesos agent, 
> launchTask, updateStatus and SendMessage2Executore to Mesos Agent 
> directly, bypass the Master?
>
> Will invoke big conflict with current mechanism?
>
>
>
> Looking forward to your comments and opinions.
>
>
>
> Best Regards,
>
> Teng
>
>
>
>
>
>
>
> Su Teng  00241668
>
>
>
> Distributed and Parallel Software Lab
>
> Huawei Technologies Co., Ltd.
>
> Email:sut...@huawei.com
>
>
>
>
>

Re: Question about "Framework directly access Meso agent"

[Guangya Liu]

For your concern of large scale cluster, there is indeed a JIRA tracking
this https://issues.apache.org/jira/browse/MESOS-3548

Thanks,

Guangya

On Wed, Feb 17, 2016 at 12:14 PM, Suteng  wrote:

> Hi,
>
>
>
> Currently, Mesos framework’s task related operations lauchTask,
> updateStatus and executorSendMessage etc., and resource related operations
> resourceOffer etc., all operations are pass through Mesos Master.
>
> When the cluster and task number become huge, or with optimistic resource
> offer, multi-framework concurrently launchTask, maybe Mesos Master will be
> a bottleneck.
>
> Is possible for framework scheduler directly access Mesos agent,
> launchTask, updateStatus and SendMessage2Executore to Mesos Agent directly,
> bypass the Master?
>
> Will invoke big conflict with current mechanism?
>
>
>
> Looking forward to your comments and opinions.
>
>
>
> Best Regards,
>
> Teng
>
>
>
>
>
>
>
> Su Teng  00241668
>
>
>
> Distributed and Parallel Software Lab
>
> Huawei Technologies Co., Ltd.
>
> Email:sut...@huawei.com
>
>
>
>
>



-- 
Guangya Liu (刘光亚)
Senior Software Engineer
DCOS and OpenStack Development
IBM Platform Computing
Systems and Technology Group

Re: Question about "Framework directly access Meso agent"

[Alex Rukletsov]

Suteng—

such optimization makes sense in certain cases (e.g. sending a framework
message), but it can be rather tricky in general, because the master has to
maintain bookkeeping. Moreover, with the upcoming HTTP API it becomes
harder for a framework to determine where to send messages to reach a
specific agent.

Have you done any performance tests and seen master becoming a bottleneck?

On Wed, Feb 17, 2016 at 5:14 AM, Suteng  wrote:

> Hi,
>
>
>
> Currently, Mesos framework’s task related operations lauchTask,
> updateStatus and executorSendMessage etc., and resource related operations
> resourceOffer etc., all operations are pass through Mesos Master.
>
> When the cluster and task number become huge, or with optimistic resource
> offer, multi-framework concurrently launchTask, maybe Mesos Master will be
> a bottleneck.
>
> Is possible for framework scheduler directly access Mesos agent,
> launchTask, updateStatus and SendMessage2Executore to Mesos Agent directly,
> bypass the Master?
>
> Will invoke big conflict with current mechanism?
>
>
>
> Looking forward to your comments and opinions.
>
>
>
> Best Regards,
>
> Teng
>
>
>
>
>
>
>
> Su Teng  00241668
>
>
>
> Distributed and Parallel Software Lab
>
> Huawei Technologies Co., Ltd.
>
> Email:sut...@huawei.com
>
>
>
>
>

Re: Question about "Framework directly access Meso agent"

[Guangya Liu]

For your concern of large scale cluster, there is indeed a JIRA tracking
this https://issues.apache.org/jira/browse/MESOS-3548

Thanks,

Guangya

On Wed, Feb 17, 2016 at 12:14 PM, Suteng  wrote:

> Hi,
>
>
>
> Currently, Mesos framework’s task related operations lauchTask,
> updateStatus and executorSendMessage etc., and resource related operations
> resourceOffer etc., all operations are pass through Mesos Master.
>
> When the cluster and task number become huge, or with optimistic resource
> offer, multi-framework concurrently launchTask, maybe Mesos Master will be
> a bottleneck.
>
> Is possible for framework scheduler directly access Mesos agent,
> launchTask, updateStatus and SendMessage2Executore to Mesos Agent directly,
> bypass the Master?
>
> Will invoke big conflict with current mechanism?
>
>
>
> Looking forward to your comments and opinions.
>
>
>
> Best Regards,
>
> Teng
>
>
>
>
>
>
>
> Su Teng  00241668
>
>
>
> Distributed and Parallel Software Lab
>
> Huawei Technologies Co., Ltd.
>
> Email:sut...@huawei.com
>
>
>
>
>



-- 
Guangya Liu (刘光亚)
Senior Software Engineer
DCOS and OpenStack Development
IBM Platform Computing
Systems and Technology Group

Question about "Framework directly access Meso agent"

[Suteng]

Hi,

Currently, Mesos framework's task related operations lauchTask, updateStatus 
and executorSendMessage etc., and resource related operations resourceOffer 
etc., all operations are pass through Mesos Master.
When the cluster and task number become huge, or with optimistic resource 
offer, multi-framework concurrently launchTask, maybe Mesos Master will be a 
bottleneck.
Is possible for framework scheduler directly access Mesos agent, launchTask, 
updateStatus and SendMessage2Executore to Mesos Agent directly, bypass the 
Master?
Will invoke big conflict with current mechanism?

Looking forward to your comments and opinions.

Best Regards,
Teng


[cid:image001.png@01D16976.1DFAFB20]


Su Teng  00241668


Distributed and Parallel Software Lab
Huawei Technologies Co., Ltd.
Email:sut...@huawei.com