Zhen - You're right, Impala does create a lot of threads, and the vast majority are in the RPC handlers. I'm going to work on reducing this number in the next couple of months - it's a little trickier than changing the Thrift server mode, unfortunately.
However, these threads are long-lived, so there's not much overhead from thread creation or destruction. The main cost is the extra memory for the stacks, and context-switching overhead. Neither show up on our profiles, but on larger clusters we do hit ulimits occasionally. Watch this space :) Henry On 26 March 2016 at 14:13, Silvius Rus <[email protected]> wrote: > [email protected], Henry, Marcel > bcc:[email protected] > > I believe this is documented as > https://issues.cloudera.org/browse/IMPALA-1514. > > Henry, Marcel, do you know if this will be worked on as part of upcoming > scalability improvements? > > On Fri, Mar 25, 2016 at 8:54 PM, zhen <[email protected]> wrote: > >> >> Recently debug impala performance found that the number of threads >> suddenly high and low >> >> >> About two cases >> >> 1. A complex query will have many fragments >> 2. big cluster example ：200 impala ins >> 3. many query running >> >> just now impala do not have thread pool >> a .ThriftServer use thread mode >> b. stream sender async channel >> will create large volumes of threads ，as everyone knows thread create >> destroy is high cost in performance！！ >> >> what is the next plan TODO >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Impala Dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > > -- Henry Robinson Software Engineer Cloudera 415-994-6679
