Hi Zhijie, Looks great that YARN is mature enough to support so large cluster. It gives me more confidence to use :)
Thanks a lot! Best, Bo On Tue, Sep 2, 2014 at 2:41 PM, Zhijie Shen <[email protected]> wrote: > Hi Bo, > > I don't have the exact number about the max concurrent job. FYI, RM is > multip-threaded, but the threads are working for different purpose. For > example, scheduler and rmstatestore has their separate thread, and RPC > calls are on individual threads as well. It's complicated to evaluate the > upper bound of concurrent apps, but I've heard of the YARN cluster > deployment on a cluster of thousands of nodes. > > Thanks, > Zhijie > > > On Tue, Sep 2, 2014 at 10:42 AM, bo yang <[email protected]> wrote: > >> Hi Zhijie, >> >> That is great to know. Thanks! >> >> So there seems no be much limit to support large concurrency. To move >> this question further, what might be the max number of concurrent jobs >> which one Resource Manager could support? Is there any numbers from your >> experience? >> >> Thanks, >> Bo >> >> >> >> >> >> >> On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <[email protected]> >> wrote: >> >>> Hi Bo, >>> >>> RM doesn't create an individual thread for each running app. The app >>> life cycle management is event driven. There's a dispatcher, which runs on >>> one thread to handle the events for all apps. >>> >>> Zhijie >>> >>> >>> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <[email protected]> wrote: >>> >>>> Hi Guys, >>>> >>>> I am thinking how many concurrent jobs a single Resource Manager might >>>> be able to manage? Following is my understanding, please correct me if I am >>>> wrong. >>>> >>>> Let's say if we have 1000 concurrent jobs running. Resource Manager >>>> will have 1000 records in memory to manage these jobs. And it will also >>>> have 1000 threads, where each thread is waiting for one job to finish. >>>> >>>> The memory part will probably be ok. For the 1000 threads, will there >>>> be any potential problem? >>>> >>>> Thanks, >>>> Bo >>>> >>> >>> >>> >>> -- >>> Zhijie Shen >>> Hortonworks Inc. >>> http://hortonworks.com/ >>> >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity >>> to which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender immediately >>> and delete it from your system. Thank You. >> >> >> > > > -- > Zhijie Shen > Hortonworks Inc. > http://hortonworks.com/ > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
