Alex, We don't have test the performance of mesos. But we have develop a framework in house, which is like a simplified mesos, use to schedule a large number fine grain computation tasks. We find that master will be a bottleneck. One reason is our task contain several KB data, and task number is quite huge. If we use mesos to replace it, maybe master still be a bottleneck.
Master still do the resource bookkeeping, we can decompose launch task to two steps, firstly scheduler tell master which offer he wants, then master tell scheduler the address of agent. Secondly, scheduler can directly launch task to the agent, and also can directly send message to agent. Maybe I can do some test about the mesos master launch task throughput, with different number task data. -----邮件原件----- 发件人: C Rukletsov [mailto:[email protected]] 发送时间: 2016年2月17日 18:04 收件人: dev 主题: Re: Question about "Framework directly access Meso agent" Suteng— such optimization makes sense in certain cases (e.g. sending a framework message), but it can be rather tricky in general, because the master has to maintain bookkeeping. Moreover, with the upcoming HTTP API it becomes harder for a framework to determine where to send messages to reach a specific agent. Have you done any performance tests and seen master becoming a bottleneck? On Wed, Feb 17, 2016 at 5:14 AM, Suteng <[email protected]> wrote: > Hi, > > > > Currently, Mesos framework’s task related operations lauchTask, > updateStatus and executorSendMessage etc., and resource related > operations resourceOffer etc., all operations are pass through Mesos Master. > > When the cluster and task number become huge, or with optimistic > resource offer, multi-framework concurrently launchTask, maybe Mesos > Master will be a bottleneck. > > Is possible for framework scheduler directly access Mesos agent, > launchTask, updateStatus and SendMessage2Executore to Mesos Agent > directly, bypass the Master? > > Will invoke big conflict with current mechanism? > > > > Looking forward to your comments and opinions. > > > > Best Regards, > > Teng > > > > > > > > Su Teng 00241668 > > > > Distributed and Parallel Software Lab > > Huawei Technologies Co., Ltd. > > Email:[email protected] > > > > >
