[ 
https://issues.apache.org/jira/browse/HBASE-19701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16367924#comment-16367924
 ] 

stack commented on HBASE-19701:
-------------------------------

Bug. To be fixed. Shouldn't block beta-2 though. Moved it out.

> Close without justification following succesful open
> ----------------------------------------------------
>
>                 Key: HBASE-19701
>                 URL: https://issues.apache.org/jira/browse/HBASE-19701
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Priority: Critical
>             Fix For: 2.0.0
>
>
> [~jmspaggi] conjured an interesting condition where we close a region soon 
> after open WITHOUT seemingly saying why (It looks like Master is asking for 
> region CLOSE but that is not clear looking at RegionServer log).
> Here is log snippet from https://pastebin.com/0r76Y6ap (in case the pastebin 
> evaporates)
> {code}
> ....
> 2017-12-31 09:54:20,864 INFO  
> [PostOpenDeployTasks:f49f3cbb7f3db4cf96c7eb3b0cf83869] 
> regionserver.HRegionServer: Post open deploy tasks for 
> TestTable,00000000000000000408944640,1505391191559.f49f3cbb7f3db4cf96c7eb3b0cf83869.
> 2017-12-31 09:54:20,870 INFO  
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1] 
> regionserver.CompactingMemStore: Setting in-memory flush size threshold to 
> 13421772 and immutable segments index to be of type CHUNK_MAP
> 2017-12-31 09:54:20,870 INFO  
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1] regionserver.HStore: 
> Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore
> 2017-12-31 09:54:20,870 INFO  
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1] hfile.CacheConfig: Created 
> cacheConfig for info: blockCache=LruBlockCache{blockCount=0, 
> currentSize=2454760, freeSize=3347745560, maxSize=3350200320, 
> heapSize=2454760, minSize=3182690304, minFactor=0.95, multiSize=1591345152, 
> multiFactor=0.5, singleSize=795672576, singleFactor=0.25}, 
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, 
> prefetchOnOpen=false
> 2017-12-31 09:54:20,872 INFO  
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1] 
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807, 
> 9223372036854775807); files [3, 10); ratio 1,200000; off-peak ratio 5,000000; 
> throttle point 2684354560; major period 604800000, major jitter 0,500000, min 
> locality to compact 0,000000; tiered compaction: max_age 9223372036854775807, 
> incoming window min 6, compaction policy for tiered window 
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, 
> single output for minor true, compaction window factory 
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2017-12-31 09:54:20,903 INFO  
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1] 
> regionserver.CompactingMemStore: Setting in-memory flush size threshold to 
> 13421772 and immutable segments index to be of type CHUNK_MAP
> 2017-12-31 09:54:20,904 INFO  
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1] regionserver.HStore: 
> Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore
> 2017-12-31 09:54:20,904 INFO  
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1] hfile.CacheConfig: Created 
> cacheConfig for info: blockCache=LruBlockCache{blockCount=0, 
> currentSize=2454760, freeSize=3347745560, maxSize=3350200320, 
> heapSize=2454760, minSize=3182690304, minFactor=0.95, multiSize=1591345152, 
> multiFactor=0.5, singleSize=795672576, singleFactor=0.25}, 
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, 
> prefetchOnOpen=false
> 2017-12-31 09:54:20,905 INFO  
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1] 
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807, 
> 9223372036854775807); files [3, 10); ratio 1,200000; off-peak ratio 5,000000; 
> throttle point 2684354560; major period 604800000, major jitter 0,500000, min 
> locality to compact 0,000000; tiered compaction: max_age 9223372036854775807, 
> incoming window min 6, compaction policy for tiered window 
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, 
> single output for minor true, compaction window factory 
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2017-12-31 09:54:20,929 INFO  [RS_OPEN_REGION-node1:16020-1] 
> regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the 
> region=TestTable,00000000000000000262144000,1505391191559.166b9c45d7724f72fd126adb4445d6ec.
> 2017-12-31 09:54:20,956 INFO  [RS_OPEN_REGION-node1:16020-0] 
> regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the 
> region=TestTable,00000000000000000188743680,1505391191559.330f09f4a0eaf26811c320fbf1b14e70.
> 2017-12-31 09:54:20,991 INFO  [RS_OPEN_REGION-node1:16020-1] 
> regionserver.HRegion: Onlined 166b9c45d7724f72fd126adb4445d6ec; next 
> sequenceid=22861
> 2017-12-31 09:54:20,998 INFO  
> [PostOpenDeployTasks:166b9c45d7724f72fd126adb4445d6ec] 
> regionserver.HRegionServer: Post open deploy tasks for 
> TestTable,00000000000000000262144000,1505391191559.166b9c45d7724f72fd126adb4445d6ec.
> 2017-12-31 09:54:21,025 INFO  [RS_OPEN_REGION-node1:16020-0] 
> regionserver.HRegion: Onlined 330f09f4a0eaf26811c320fbf1b14e70; next 
> sequenceid=21870
> 2017-12-31 09:54:21,031 INFO  
> [PostOpenDeployTasks:330f09f4a0eaf26811c320fbf1b14e70] 
> regionserver.HRegionServer: Post open deploy tasks for 
> TestTable,00000000000000000188743680,1505391191559.330f09f4a0eaf26811c320fbf1b14e70.
> 2017-12-31 09:54:21,036 INFO  
> [StoreOpener-ba17cd27e2822b91193359a273684ac2-1] 
> regionserver.CompactingMemStore: Setting in-memory flush size threshold to 
> 13421772 and immutable segments index to be of type CHUNK_MAP
> 2017-12-31 09:54:21,036 INFO  
> [StoreOpener-ba17cd27e2822b91193359a273684ac2-1] regionserver.HStore: 
> Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore
> 2017-12-31 09:54:21,036 INFO  
> [StoreOpener-ba17cd27e2822b91193359a273684ac2-1] hfile.CacheConfig: Created 
> cacheConfig for info: blockCache=LruBlockCache{blockCount=0, 
> currentSize=2454760, freeSize=3347745560, maxSize=3350200320, 
> heapSize=2454760, minSize=3182690304, minFactor=0.95, multiSize=1591345152, 
> multiFactor=0.5, singleSize=795672576, singleFactor=0.25}, 
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, 
> prefetchOnOpen=false
> 2017-12-31 09:54:21,037 INFO  
> [StoreOpener-ba17cd27e2822b91193359a273684ac2-1] 
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807, 
> 9223372036854775807); files [3, 10); ratio 1,200000; off-peak ratio 5,000000; 
> throttle point 2684354560; major period 604800000, major jitter 0,500000, min 
> locality to compact 0,000000; tiered compaction: max_age 9223372036854775807, 
> incoming window min 6, compaction policy for tiered window 
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, 
> single output for minor true, compaction window factory 
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2017-12-31 09:54:21,058 INFO  
> [StoreOpener-e6ad456daff7a3d4ad259eab64a61c11-1] 
> regionserver.CompactingMemStore: Setting in-memory flush size threshold to 
> 13421772 and immutable segments index to be of type CHUNK_MAP
> 2017-12-31 09:54:21,059 INFO  
> [StoreOpener-e6ad456daff7a3d4ad259eab64a61c11-1] regionserver.HStore: 
> Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore
> 2017-12-31 09:54:21,059 INFO  
> [StoreOpener-e6ad456daff7a3d4ad259eab64a61c11-1] hfile.CacheConfig: Created 
> cacheConfig for info: blockCache=LruBlockCache{blockCount=0, 
> currentSize=2454760, freeSize=3347745560, maxSize=3350200320, 
> heapSize=2454760, minSize=3182690304, minFactor=0.95, multiSize=1591345152, 
> multiFactor=0.5, singleSize=795672576, singleFactor=0.25}, 
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, 
> prefetchOnOpen=false
> 2017-12-31 09:54:21,063 INFO  
> [StoreOpener-e6ad456daff7a3d4ad259eab64a61c11-1] 
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807, 
> 9223372036854775807); files [3, 10); ratio 1,200000; off-peak ratio 5,000000; 
> throttle point 2684354560; major period 604800000, major jitter 0,500000, min 
> locality to compact 0,000000; tiered compaction: max_age 9223372036854775807, 
> incoming window min 6, compaction policy for tiered window 
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, 
> single output for minor true, compaction window factory 
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2017-12-31 09:54:21,150 INFO  [RS_OPEN_REGION-node1:16020-2] 
> regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the 
> region=TestTable,00000000000000000555745280,1505391191559.19d79b967c2e59cef288874533c154c8.
> 2017-12-31 09:54:21,234 INFO  [RS_OPEN_REGION-node1:16020-0] 
> regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the 
> region=TestTable,00000000000000000115343360,1505391191559.e6ad456daff7a3d4ad259eab64a61c11.
> 2017-12-31 09:54:21,258 INFO  [RS_OPEN_REGION-node1:16020-2] 
> regionserver.HRegion: Onlined 19d79b967c2e59cef288874533c154c8; next 
> sequenceid=23352
> 2017-12-31 09:54:21,264 INFO  
> [PostOpenDeployTasks:19d79b967c2e59cef288874533c154c8] 
> regionserver.HRegionServer: Post open deploy tasks for 
> TestTable,00000000000000000555745280,1505391191559.19d79b967c2e59cef288874533c154c8.
> 2017-12-31 09:54:21,313 INFO  [RS_OPEN_REGION-node1:16020-0] 
> regionserver.HRegion: Onlined e6ad456daff7a3d4ad259eab64a61c11; next 
> sequenceid=21294
> 2017-12-31 09:54:21,321 INFO  [RS_OPEN_REGION-node1:16020-1] 
> regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the 
> region=TestTable,00000000000000000335544320,1505391191559.ba17cd27e2822b91193359a273684ac2.
> 2017-12-31 09:54:21,321 INFO  
> [PostOpenDeployTasks:e6ad456daff7a3d4ad259eab64a61c11] 
> regionserver.HRegionServer: Post open deploy tasks for 
> TestTable,00000000000000000115343360,1505391191559.e6ad456daff7a3d4ad259eab64a61c11.
> 2017-12-31 09:54:21,380 INFO  [RS_OPEN_REGION-node1:16020-1] 
> regionserver.HRegion: Onlined ba17cd27e2822b91193359a273684ac2; next 
> sequenceid=22282
> 2017-12-31 09:54:21,386 INFO  
> [PostOpenDeployTasks:ba17cd27e2822b91193359a273684ac2] 
> regionserver.HRegionServer: Post open deploy tasks for 
> TestTable,00000000000000000335544320,1505391191559.ba17cd27e2822b91193359a273684ac2.
> 2017-12-31 09:54:40,649 INFO  
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> regionserver.RSRpcServices: Close 166b9c45d7724f72fd126adb4445d6ec without 
> moving
> 2017-12-31 09:54:40,655 INFO  
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> regionserver.RSRpcServices: Close ba17cd27e2822b91193359a273684ac2 without 
> moving
> 2017-12-31 09:54:40,680 INFO  
> [StoreCloserThread-TestTable,00000000000000000335544320,1505391191559.ba17cd27e2822b91193359a273684ac2.-1]
>  regionserver.HStore: Closed info
> 2017-12-31 09:54:40,680 INFO  
> [StoreCloserThread-TestTable,00000000000000000262144000,1505391191559.166b9c45d7724f72fd126adb4445d6ec.-1]
>  regionserver.HStore: Closed info
> 2017-12-31 09:54:40,702 INFO  
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> regionserver.RSRpcServices: Close 98025e37220dbc821f862a91d05a96ef without 
> moving
> 2017-12-31 09:54:40,703 INFO  
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> regionserver.RSRpcServices: Close 19d79b967c2e59cef288874533c154c8 without 
> moving
> 2017-12-31 09:54:40,707 INFO  
> [StoreCloserThread-TestTable,00000000000000000482344960,1505391191559.98025e37220dbc821f862a91d05a96ef.-1]
>  regionserver.HStore: Closed info
> 2017-12-31 09:54:40,741 INFO  [RS_CLOSE_REGION-node1:16020-0] 
> regionserver.HRegion: Closed 
> TestTable,00000000000000000262144000,1505391191559.166b9c45d7724f72fd126adb4445d6ec.
> 2017-12-31 09:54:40,754 INFO  
> [StoreCloserThread-TestTable,00000000000000000555745280,1505391191559.19d79b967c2e59cef288874533c154c8.-1]
>  regionserver.HStore: Closed info
> 2017-12-31 09:54:40,762 INFO  [RS_CLOSE_REGION-node1:16020-1] 
> regionserver.HRegion: Closed 
> TestTable,00000000000000000335544320,1505391191559.ba17cd27e2822b91193359a273684ac2.
> 2017-12-31 09:54:40,807 INFO  [RS_CLOSE_REGION-node1:16020-2] 
> regionserver.HRegion: Closed 
> TestTable,00000000000000000482344960,1505391191559.98025e37220dbc821f862a91d05a96ef.
> 2017-12-31 09:54:40,839 INFO  [RS_CLOSE_REGION-node1:16020-0] 
> regionserver.HRegion: Closed 
> TestTable,00000000000000000555745280,1505391191559.19d79b967c2e59cef288874533c154c8.
> 2017-12-31 09:54:41,102 INFO  
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> regionserver.RSRpcServices: Close f49f3cbb7f3db4cf96c7eb3b0cf83869 without 
> moving
> 2017-12-31 09:54:41,105 INFO  
> [StoreCloserThread-TestTable,00000000000000000408944640,1505391191559.f49f3cbb7f3db4cf96c7eb3b0cf83869.-1]
>  regionserver.HStore: Closed info
> 2017-12-31 09:54:41,184 INFO  [RS_CLOSE_REGION-node1:16020-1] 
> regionserver.HRegion: Closed 
> TestTable,00000000000000000408944640,1505391191559.f49f3cbb7f3db4cf96c7eb3b0cf83869.
> 2017-12-31 09:54:42,249 INFO  
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> regionserver.RSRpcServices: Close e6ad456daff7a3d4ad259eab64a61c11 without 
> moving
> 2017-12-31 09:54:42,250 INFO  
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> regionserver.RSRpcServices: Close 330f09f4a0eaf26811c320fbf1b14e70 without 
> moving
> 2017-12-31 09:54:42,250 INFO  
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> regionserver.RSRpcServices: Close 655b0a0608ebaecacfda2e25a7e4b0f5 without 
> moving
> 2017-12-31 09:54:42,253 INFO  
> [StoreCloserThread-TestTable,00000000000000000115343360,1505391191559.e6ad456daff7a3d4ad259eab64a61c11.-1]
>  regionserver.HStore: Closed info
> 2017-12-31 09:54:42,254 INFO  
> [StoreCloserThread-TestTable,00000000000000000188743680,1505391191559.330f09f4a0eaf26811c320fbf1b14e70.-1]
>  regionserver.HStore: Closed info
> 2017-12-31 09:54:42,254 INFO  
> [StoreCloserThread-TestTable,00000000000000000178257920,1505391191559.655b0a0608ebaecacfda2e25a7e4b0f5.-1]
>  regionserver.HStore: Closed info
> 2017-12-31 09:54:42,341 INFO  [RS_CLOSE_REGION-node1:16020-1] 
> regionserver.HRegion: Closed 
> TestTable,00000000000000000178257920,1505391191559.655b0a0608ebaecacfda2e25a7e4b0f5.
> 2017-12-31 09:54:42,360 INFO  [RS_CLOSE_REGION-node1:16020-2] 
> regionserver.HRegion: Closed 
> TestTable,00000000000000000115343360,1505391191559.e6ad456daff7a3d4ad259eab64a61c11.
> 2017-12-31 09:54:42,361 INFO  [RS_CLOSE_REGION-node1:16020-0] 
> regionserver.HRegion: Closed 
> TestTable,00000000000000000188743680,1505391191559.330f09f4a0eaf26811c320fbf1b14e70.
> 2017-12-31 09:54:42,412 INFO  
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] 
> regionserver.RSRpcServices: Close 30eaa72ddc1f2706dc84c62e3222a006 without 
> moving
> 2017-12-31 09:54:42,415 INFO  
> [StoreCloserThread-TestTable,00000000000000000041943040,1505391191559.30eaa72ddc1f2706dc84c62e3222a006.-1]
>  regionserver.HStore: Closed info
> 2017-12-31 09:54:42,495 INFO  [RS_CLOSE_REGION-node1:16020-1] 
> regionserver.HRegion: Closed 
> TestTable,00000000000000000041943040,1505391191559.30eaa72ddc1f2706dc84c62e3222a006.
> 2017-12-31 09:54:53,069 INFO  [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient: 
> 0x1df82317 no activities for 60000 ms, close active connection. Will 
> reconnect next time when there are new requests.
> 2017-12-31 09:54:53,088 INFO  [ReadOnlyZKClient] zookeeper.ZooKeeper: 
> Session: 0x160a94d45ab0122 closed
> 2017-12-31 09:54:53,088 INFO  [ReadOnlyZKClient-EventThread] 
> zookeeper.ClientCnxn: EventThread shut down for session: 0x160a94d45ab0122
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to