We have no regions in transition. As far as who's requesting the table descriptors: We are calling HTableIterface.getTableDescriptor frequently. We are going to cache the result to avoid the unnecessary network calls.
Thanks for the hint! From: lars hofhansl [mailto:[email protected]] Sent: Thursday, March 26, 2015 3:43 PM To: [email protected] Cc: Development Subject: Re: master consumes large amount of CPU for days Hi Ted, Yes, it is safe to bounce the HMaster without taking the region servers down. Are there any regions in transition (would be shown on the master's page)? All of the threads involved with - org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(org.apache.hadoop.fs.Path) @bci=14, line=309 (Compiled frame) ... - org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(java.util.List) @bci=50, line=1932 (Compiled frame) are a bit suspect. Who's requesting all the table descriptors from the master? -- Lars ________________________________ From: Ted Tuttle <[email protected]<mailto:[email protected]>> To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Cc: Development <[email protected]<mailto:[email protected]>> Sent: Thursday, March 26, 2015 12:18 PM Subject: master consumes large amount of CPU for days Hello- Our master process started consuming a large amount of CPU (75% of box) several days back and hasn't stopped. I have 2 questions: 1) what is it doing (stack dump and log below) 2) is it safe to restart the master without taking the whole cluster down? Master stack dump: http://pastebin.com/G0iNNEpC Master log from last 15 mins or so: http://pastebin.com/WQNjhFGf Thanks, Ted
