Hi Bo, I don't have the exact number about the max concurrent job. FYI, RM is multip-threaded, but the threads are working for different purpose. For example, scheduler and rmstatestore has their separate thread, and RPC calls are on individual threads as well. It's complicated to evaluate the upper bound of concurrent apps, but I've heard of the YARN cluster deployment on a cluster of thousands of nodes.
Thanks, Zhijie On Tue, Sep 2, 2014 at 10:42 AM, bo yang <[email protected]> wrote: > Hi Zhijie, > > That is great to know. Thanks! > > So there seems no be much limit to support large concurrency. To move this > question further, what might be the max number of concurrent jobs which one > Resource Manager could support? Is there any numbers from your experience? > > Thanks, > Bo > > > > > > > On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <[email protected]> > wrote: > >> Hi Bo, >> >> RM doesn't create an individual thread for each running app. The app life >> cycle management is event driven. There's a dispatcher, which runs on one >> thread to handle the events for all apps. >> >> Zhijie >> >> >> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <[email protected]> wrote: >> >>> Hi Guys, >>> >>> I am thinking how many concurrent jobs a single Resource Manager might >>> be able to manage? Following is my understanding, please correct me if I am >>> wrong. >>> >>> Let's say if we have 1000 concurrent jobs running. Resource Manager will >>> have 1000 records in memory to manage these jobs. And it will also have >>> 1000 threads, where each thread is waiting for one job to finish. >>> >>> The memory part will probably be ok. For the 1000 threads, will there be >>> any potential problem? >>> >>> Thanks, >>> Bo >>> >> >> >> >> -- >> Zhijie Shen >> Hortonworks Inc. >> http://hortonworks.com/ >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. > > > -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
