[
https://issues.apache.org/jira/browse/HBASE-555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584269#action_12584269
]
stack commented on HBASE-555:
-----------------------------
This patch should include upper bound on how many regions we assign at a time.
Here is what poor old server .15 gets when he innocently reports in for duty:
{code}
2008-04-01 00:06:58,851 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20070223009_20070927,1205860531876 to server
192.168.105.19:600202008-04-01 00:06:58,853 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20070176389_20070802,1205827644321 to server
192.168.105.19:600202008-04-01 00:06:58,853 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5980569_19991109,1205804361124 to server
192.168.105.19:600202008-04-01 00:06:58,853 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5039934_19910813,1205720875870 to server
192.168.105.19:600202008-04-01 00:06:58,853 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6294754_20010925,1205782333675 to server
192.168.105.19:600202008-04-01 00:06:58,853 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6263923_20010724,1206042922873 to server
192.168.105.19:600202008-04-01 00:06:58,854 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5932059_19990803,1205808352696 to server
192.168.105.19:600202008-04-01 00:06:58,854 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,EP00940773NWA1,1206819926889 to server 192.168.105.19:600202008-04-01
00:06:58,854 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20070070734_20070329,1205797891897 to server
192.168.105.19:600202008-04-01 00:06:58,854 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6651733_20031125,1205799265993 to server
192.168.105.19:600202008-04-01 00:06:58,854 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20030084507_20030508,1205783548816 to server
192.168.105.19:600202008-04-01 00:06:58,854 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5750981_19980512,1205695699495 to server
192.168.105.19:600202008-04-01 00:06:58,854 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20040244841_20041209,1205813956207 to server
192.168.105.19:600202008-04-01 00:06:58,854 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6341961_20020129,1205838276676 to server
192.168.105.19:600202008-04-01 00:06:58,855 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20040178030_20040916,1205745073891 to server
192.168.105.19:600202008-04-01 00:06:58,855 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20070053028_20070308,1205799199459 to server
192.168.105.19:600202008-04-01 00:06:58,855 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5956468_19990921,1205807569500 to server
192.168.105.19:600202008-04-01 00:06:58,855 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6273754_20010814,1205780845479 to server
192.168.105.19:600202008-04-01 00:06:58,855 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20040027221_20040212,1205768933859 to server
192.168.105.19:600202008-04-01 00:06:58,855 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20020168470_20021114,1205839625070 to server
192.168.105.19:600202008-04-01 00:06:58,855 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6433670_20020813,1205883860520 to server
192.168.105.19:600202008-04-01 00:06:58,855 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20080016321_20080117,1205710780019 to server
192.168.105.19:600202008-04-01 00:06:58,855 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20020193527_20021219,1205835604483 to server
192.168.105.19:600202008-04-01 00:06:58,856 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,EP95306226NWA1,1206744544751 to server 192.168.105.19:600202008-04-01
00:06:58,856 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,EP97305470NWA2,1206746225615 to server 192.168.105.19:600202008-04-01
00:06:58,856 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20010053170_20011220,1205853437060 to server
192.168.105.19:600202008-04-01 00:06:58,856 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20050183021_20050818,1205844147766 to server
192.168.105.19:600202008-04-01 00:06:58,856 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20020155447_20021024,1205837665094 to server
192.168.105.19:600202008-04-01 00:06:58,856 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5781595_19980714,1205694512635 to server
192.168.105.19:600202008-04-01 00:06:58,856 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,EP00303162NWA1,1206745983715 to server 192.168.105.19:600202008-04-01
00:06:58,856 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20010047773_20011206,1205853775465 to server
192.168.105.19:600202008-04-01 00:06:58,856 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5485829_19960123,1205788999315 to server
192.168.105.19:600202008-04-01 00:06:58,857 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6457870_20021001,1205883490706 to server
192.168.105.19:600202008-04-01 00:06:58,857 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20030036046_20030220,1205786541377 to server
192.168.105.19:600202008-04-01 00:06:58,857 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5381387_19950110,1205758160021 to server
192.168.105.19:600202008-04-01 00:06:58,857 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20040226216_20041118,1205814276434 to server
192.168.105.19:600202008-04-01 00:06:58,857 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5093930_19920303,1205714978791 to server
192.168.105.19:600202008-04-01 00:06:58,857 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20020142901_20021003,1205834554230 to server
192.168.105.19:600202008-04-01 00:06:58,857 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20080023488_20080131,1205705334720 to server
192.168.105.19:600202008-04-01 00:06:58,857 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,EP94107781NWA1,1206821060442 to server 192.168.105.19:600202008-04-01
00:06:58,857 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20020161344_20021031,1205837872228 to server
192.168.105.19:600202008-04-01 00:06:58,858 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20070049359_20070301,1205799262064 to server
192.168.105.19:600202008-04-01 00:06:58,858 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20060132645_20060622,1205723325691 to server
192.168.105.19:600202008-04-01 00:06:58,858 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6944277_20050913,1205811736641 to server
192.168.105.19:600202008-04-01 00:06:58,858 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,EP05251320NWA2,1206745588948 to server 192.168.105.19:600202008-04-01
00:06:58,858 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20050045317_20050303,1205775329018 to server
192.168.105.19:600202008-04-01 00:06:58,858 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20050015798_20050120,1205775930084 to server
192.168.105.19:600202008-04-01 00:06:58,858 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5586783_19961224,1205768599392 to server
192.168.105.19:600202008-04-01 00:06:58,858 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5951449_19990914,1205805748422 to server
192.168.105.19:600202008-04-01 00:06:58,858 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6074923_20000613,1205828088350 to server
192.168.105.19:600202008-04-01 00:06:58,859 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6874596_20050405,1205873222614 to server
192.168.105.19:600202008-04-01 00:06:58,859 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6334752_20020101,1205834669245 to server
192.168.105.19:600202008-04-01 00:06:58,859 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20020132259_20020919,1205835048047 to server
192.168.105.19:600202008-04-01 00:06:58,859 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,EP99923891NWA1,1206819796374 to server 192.168.105.19:600202008-04-01
00:06:58,859 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20050033656_20050210,1205774486855 to server
192.168.105.19:600202008-04-01 00:06:58,859 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6901250_20050531,1205838762335 to server
192.168.105.19:600202008-04-01 00:06:58,859 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20030162836_20030828,1205729556792 to server
192.168.105.19:600202008-04-01 00:06:58,859 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5964518_19991012,1205807734920 to server
192.168.105.19:600202008-04-01 00:06:58,859 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20050021366_20050127,1205776780745 to server
192.168.105.19:600202008-04-01 00:06:58,860 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20070225042_20070927,1205858223863 to server
192.168.105.19:600202008-04-01 00:06:58,860 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5368612_19941129,1205728257271 to server
192.168.105.19:600202008-04-01 00:06:58,860 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20050237098_20051027,1205752084253 to server 192.168.105.19:60020
2008-04-01 00:06:58,860 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6252229_20010626,1205783728462 to server
192.168.105.19:600202008-04-01 00:06:58,860 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6501540_20021231,1205864683038 to server
192.168.105.19:600202008-04-01 00:06:58,860 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6088180_20000711,1205826753254 to server
192.168.105.19:600202008-04-01 00:06:58,860 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5717290_19980210,1205695728836 to server
192.168.105.19:600202008-04-01 00:06:58,860 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,EP99905280NWA1,1206819796374 to server 192.168.105.19:600202008-04-01
00:06:58,861 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20030006741_20030109,1205786460019 to server
192.168.105.19:600202008-04-01 00:06:58,861 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US7227408_20070605,1205693336841 to server
192.168.105.19:600202008-04-01 00:06:58,861 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20070191249_20070816,1205827681299 to server
192.168.105.19:600202008-04-01 00:06:58,861 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5726633_19980310,1205695056643 to server
192.168.105.19:600202008-04-01 00:06:58,861 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US4421936_19831220,1205691633404 to server
192.168.105.19:600202008-04-01 00:06:58,861 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6883063_20050419,1205876207585 to server
192.168.105.19:600202008-04-01 00:06:58,861 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6993367_20060131,1205812286647 to server
192.168.105.19:600202008-04-01 00:06:58,861 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20040032680_20040219,1205767506895 to server
192.168.105.19:600202008-04-01 00:06:58,862 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5074216_19911224,1205714927547 to server
192.168.105.19:600202008-04-01 00:06:58,862 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20070288792_20071213,1205859526643 to server
192.168.105.19:600202008-04-01 00:06:58,862 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US4606287_19860819,1205675999782 to server
192.168.105.19:600202008-04-01 00:06:58,862 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5340031_19940823,1205726837938 to server
192.168.105.19:600202008-04-01 00:06:58,862 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5426911_19950627,1205790317865 to server
192.168.105.19:600202008-04-01 00:06:58,862 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6055976_20000502,1205828098128 to server
192.168.105.19:600202008-04-01 00:06:58,862 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6728058_20040427,1205855011163 to server
192.168.105.19:600202008-04-01 00:06:58,862 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20060198313_20060907,1205751115734 to server
192.168.105.19:600202008-04-01 00:06:58,862 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6104363_20000815,1205793128469 to server
192.168.105.19:600202008-04-01 00:06:58,863 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20070213898_20070913,1205862269967 to server
192.168.105.19:600202008-04-01 00:06:58,863 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,EP96901122NWA1,1206819790225 to server 192.168.105.19:600202008-04-01
00:06:58,863 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5029810_19910709,1205720959956 to server
192.168.105.19:600202008-04-01 00:06:58,863 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5816096_19981006,1205709691980 to server
192.168.105.19:600202008-04-01 00:06:58,863 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US7146851_20061212,1205679297543 to server
192.168.105.19:600202008-04-01 00:06:58,863 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20030088471_20030508,1205788707432 to server
192.168.105.19:600202008-04-01 00:06:58,863 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US4495261_19850122,1205690337442 to server
192.168.105.19:600202008-04-01 00:06:58,863 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US7183766_20070227,1205681384269 to server
192.168.105.19:600202008-04-01 00:06:58,864 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20020053547_20020509,1205810627716 to server
192.168.105.19:600202008-04-01 00:06:58,864 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US4672707_19870616,1205674955927 to server
192.168.105.19:600202008-04-01 00:06:58,864 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6702864_20040309,1205855019624 to server
192.168.105.19:600202008-04-01 00:06:58,864 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,EP97307534NWA2,1206746225615 to server 192.168.105.19:600202008-04-01
00:06:58,864 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20060233634_20061019,1205760231971 to server
192.168.105.19:600202008-04-01 00:06:58,864 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,EP94201405NWA2,1205775922023 to server 192.168.105.19:600202008-04-01
00:06:58,864 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20040152203_20040805,1205745590108 to server
192.168.105.19:600202008-04-01 00:06:58,864 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20020073831_20020620,1205807980345 to server
192.168.105.19:600202008-04-01 00:06:58,864 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20050097179_20050505,1205777885543 to server
192.168.105.19:600202008-04-01 00:06:58,865 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20070255107_20071101,1205858558826 to server
192.168.105.19:600202008-04-01 00:06:58,865 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6543920_20030408,1205868417155 to server
192.168.105.19:600202008-04-01 00:06:58,865 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20050146279_20050707,1205850390258 to server
192.168.105.19:600202008-04-01 00:06:58,865 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6404147_20020611,1205882087032 to server
192.168.105.19:600202008-04-01 00:06:58,865 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US7271671_20070918,1205692997730 to server
192.168.105.19:600202008-04-01 00:06:58,865 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20060252822_20061109,1205755072012 to server
192.168.105.19:600202008-04-01 00:06:58,865 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5490702_19960213,1205788999315 to server
192.168.105.19:600202008-04-01 00:06:58,865 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6210006_20010403,1205780665633 to server
192.168.105.19:600202008-04-01 00:06:58,865 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5090537_19920225,1205720385674 to server
192.168.105.19:600202008-04-01 00:06:58,866 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20050165964_20050728,1205845409575 to server
192.168.105.19:600202008-04-01 00:06:58,866 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US5760377_19980602,1205692663986 to server
192.168.105.19:600202008-04-01 00:06:58,866 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20020154020_20021024,1205837665094 to server
192.168.105.19:600202008-04-01 00:06:58,866 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6838426_20050104,1205873519309 to server
192.168.105.19:600202008-04-01 00:06:58,866 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US6630043_20031007,1205800273625 to server
192.168.105.19:600202008-04-01 00:06:58,866 INFO
org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,US20070000242_20070104,1205799280126 to server 192.168.105.19:60020
{code}
A hundred plus regions...
> Only one Worker in HRS; on startup, if assigned tens of regions, havoc of
> reassignments because open processing is done in series
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-555
> URL: https://issues.apache.org/jira/browse/HBASE-555
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.16.0, 0.2.0, 0.1.0
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Attachments: 555-0.1.patch
>
>
> On the Lars clusters, he's up into the thousands of regions. Starting this
> cluster, there is a load of churn in the master log as we assign regions,
> they report their opening and then after the hbase.hbasemaster.maxregionopen
> of one minute elapses, we assign the region elsewhere.
> Problem seems to be the fact that we only run a single Worker thread in our
> regionserver; means that region opens are processed in series.
> For example, the below shows when a master assigned a region and then the
> regionserver side log when it got around to opening it:
> {code}
> 2008-03-29 04:48:51,638 INFO org.apache.hadoop.hbase.HMaster: assigning
> region pdc-docs,US20060158177_20060720,1205765009844 to server
> 192.168.105.19:60020
> ..
> 2008-03-29 04:50:58,124 INFO org.apache.hadoop.hbase.HRegionServer:
> MSG_REGION_OPEN : pdc-docs,US20060158177_20060720,1205765009844
> {code}
> There is > 2 minutes between the two loggings (I checked clocks on this
> cluster and they are synced).
> Looking in the regionserver log, its just filled with logging on the opening
> of regions. The region opens are running pretty fast at about a second each
> but there are hundreds of regions to open in this case so its easy to go over
> our default of 60 seconds.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
