[
https://issues.apache.org/jira/browse/HBASE-19701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-19701:
--------------------------
Fix Version/s: (was: 2.0.0)
3.0.0
> Close without justification following succesful open
> ----------------------------------------------------
>
> Key: HBASE-19701
> URL: https://issues.apache.org/jira/browse/HBASE-19701
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Fix For: 3.0.0
>
>
> [~jmspaggi] conjured an interesting condition where we close a region soon
> after open WITHOUT seemingly saying why (It looks like Master is asking for
> region CLOSE but that is not clear looking at RegionServer log).
> Here is log snippet from https://pastebin.com/0r76Y6ap (in case the pastebin
> evaporates)
> {code}
> ....
> 2017-12-31 09:54:20,864 INFO
> [PostOpenDeployTasks:f49f3cbb7f3db4cf96c7eb3b0cf83869]
> regionserver.HRegionServer: Post open deploy tasks for
> TestTable,00000000000000000408944640,1505391191559.f49f3cbb7f3db4cf96c7eb3b0cf83869.
> 2017-12-31 09:54:20,870 INFO
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1]
> regionserver.CompactingMemStore: Setting in-memory flush size threshold to
> 13421772 and immutable segments index to be of type CHUNK_MAP
> 2017-12-31 09:54:20,870 INFO
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1] regionserver.HStore:
> Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore
> 2017-12-31 09:54:20,870 INFO
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1] hfile.CacheConfig: Created
> cacheConfig for info: blockCache=LruBlockCache{blockCount=0,
> currentSize=2454760, freeSize=3347745560, maxSize=3350200320,
> heapSize=2454760, minSize=3182690304, minFactor=0.95, multiSize=1591345152,
> multiFactor=0.5, singleSize=795672576, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2017-12-31 09:54:20,872 INFO
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1]
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807,
> 9223372036854775807); files [3, 10); ratio 1,200000; off-peak ratio 5,000000;
> throttle point 2684354560; major period 604800000, major jitter 0,500000, min
> locality to compact 0,000000; tiered compaction: max_age 9223372036854775807,
> incoming window min 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2017-12-31 09:54:20,903 INFO
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1]
> regionserver.CompactingMemStore: Setting in-memory flush size threshold to
> 13421772 and immutable segments index to be of type CHUNK_MAP
> 2017-12-31 09:54:20,904 INFO
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1] regionserver.HStore:
> Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore
> 2017-12-31 09:54:20,904 INFO
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1] hfile.CacheConfig: Created
> cacheConfig for info: blockCache=LruBlockCache{blockCount=0,
> currentSize=2454760, freeSize=3347745560, maxSize=3350200320,
> heapSize=2454760, minSize=3182690304, minFactor=0.95, multiSize=1591345152,
> multiFactor=0.5, singleSize=795672576, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2017-12-31 09:54:20,905 INFO
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1]
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807,
> 9223372036854775807); files [3, 10); ratio 1,200000; off-peak ratio 5,000000;
> throttle point 2684354560; major period 604800000, major jitter 0,500000, min
> locality to compact 0,000000; tiered compaction: max_age 9223372036854775807,
> incoming window min 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2017-12-31 09:54:20,929 INFO [RS_OPEN_REGION-node1:16020-1]
> regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the
> region=TestTable,00000000000000000262144000,1505391191559.166b9c45d7724f72fd126adb4445d6ec.
> 2017-12-31 09:54:20,956 INFO [RS_OPEN_REGION-node1:16020-0]
> regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the
> region=TestTable,00000000000000000188743680,1505391191559.330f09f4a0eaf26811c320fbf1b14e70.
> 2017-12-31 09:54:20,991 INFO [RS_OPEN_REGION-node1:16020-1]
> regionserver.HRegion: Onlined 166b9c45d7724f72fd126adb4445d6ec; next
> sequenceid=22861
> 2017-12-31 09:54:20,998 INFO
> [PostOpenDeployTasks:166b9c45d7724f72fd126adb4445d6ec]
> regionserver.HRegionServer: Post open deploy tasks for
> TestTable,00000000000000000262144000,1505391191559.166b9c45d7724f72fd126adb4445d6ec.
> 2017-12-31 09:54:21,025 INFO [RS_OPEN_REGION-node1:16020-0]
> regionserver.HRegion: Onlined 330f09f4a0eaf26811c320fbf1b14e70; next
> sequenceid=21870
> 2017-12-31 09:54:21,031 INFO
> [PostOpenDeployTasks:330f09f4a0eaf26811c320fbf1b14e70]
> regionserver.HRegionServer: Post open deploy tasks for
> TestTable,00000000000000000188743680,1505391191559.330f09f4a0eaf26811c320fbf1b14e70.
> 2017-12-31 09:54:21,036 INFO
> [StoreOpener-ba17cd27e2822b91193359a273684ac2-1]
> regionserver.CompactingMemStore: Setting in-memory flush size threshold to
> 13421772 and immutable segments index to be of type CHUNK_MAP
> 2017-12-31 09:54:21,036 INFO
> [StoreOpener-ba17cd27e2822b91193359a273684ac2-1] regionserver.HStore:
> Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore
> 2017-12-31 09:54:21,036 INFO
> [StoreOpener-ba17cd27e2822b91193359a273684ac2-1] hfile.CacheConfig: Created
> cacheConfig for info: blockCache=LruBlockCache{blockCount=0,
> currentSize=2454760, freeSize=3347745560, maxSize=3350200320,
> heapSize=2454760, minSize=3182690304, minFactor=0.95, multiSize=1591345152,
> multiFactor=0.5, singleSize=795672576, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2017-12-31 09:54:21,037 INFO
> [StoreOpener-ba17cd27e2822b91193359a273684ac2-1]
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807,
> 9223372036854775807); files [3, 10); ratio 1,200000; off-peak ratio 5,000000;
> throttle point 2684354560; major period 604800000, major jitter 0,500000, min
> locality to compact 0,000000; tiered compaction: max_age 9223372036854775807,
> incoming window min 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2017-12-31 09:54:21,058 INFO
> [StoreOpener-e6ad456daff7a3d4ad259eab64a61c11-1]
> regionserver.CompactingMemStore: Setting in-memory flush size threshold to
> 13421772 and immutable segments index to be of type CHUNK_MAP
> 2017-12-31 09:54:21,059 INFO
> [StoreOpener-e6ad456daff7a3d4ad259eab64a61c11-1] regionserver.HStore:
> Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore
> 2017-12-31 09:54:21,059 INFO
> [StoreOpener-e6ad456daff7a3d4ad259eab64a61c11-1] hfile.CacheConfig: Created
> cacheConfig for info: blockCache=LruBlockCache{blockCount=0,
> currentSize=2454760, freeSize=3347745560, maxSize=3350200320,
> heapSize=2454760, minSize=3182690304, minFactor=0.95, multiSize=1591345152,
> multiFactor=0.5, singleSize=795672576, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2017-12-31 09:54:21,063 INFO
> [StoreOpener-e6ad456daff7a3d4ad259eab64a61c11-1]
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807,
> 9223372036854775807); files [3, 10); ratio 1,200000; off-peak ratio 5,000000;
> throttle point 2684354560; major period 604800000, major jitter 0,500000, min
> locality to compact 0,000000; tiered compaction: max_age 9223372036854775807,
> incoming window min 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2017-12-31 09:54:21,150 INFO [RS_OPEN_REGION-node1:16020-2]
> regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the
> region=TestTable,00000000000000000555745280,1505391191559.19d79b967c2e59cef288874533c154c8.
> 2017-12-31 09:54:21,234 INFO [RS_OPEN_REGION-node1:16020-0]
> regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the
> region=TestTable,00000000000000000115343360,1505391191559.e6ad456daff7a3d4ad259eab64a61c11.
> 2017-12-31 09:54:21,258 INFO [RS_OPEN_REGION-node1:16020-2]
> regionserver.HRegion: Onlined 19d79b967c2e59cef288874533c154c8; next
> sequenceid=23352
> 2017-12-31 09:54:21,264 INFO
> [PostOpenDeployTasks:19d79b967c2e59cef288874533c154c8]
> regionserver.HRegionServer: Post open deploy tasks for
> TestTable,00000000000000000555745280,1505391191559.19d79b967c2e59cef288874533c154c8.
> 2017-12-31 09:54:21,313 INFO [RS_OPEN_REGION-node1:16020-0]
> regionserver.HRegion: Onlined e6ad456daff7a3d4ad259eab64a61c11; next
> sequenceid=21294
> 2017-12-31 09:54:21,321 INFO [RS_OPEN_REGION-node1:16020-1]
> regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the
> region=TestTable,00000000000000000335544320,1505391191559.ba17cd27e2822b91193359a273684ac2.
> 2017-12-31 09:54:21,321 INFO
> [PostOpenDeployTasks:e6ad456daff7a3d4ad259eab64a61c11]
> regionserver.HRegionServer: Post open deploy tasks for
> TestTable,00000000000000000115343360,1505391191559.e6ad456daff7a3d4ad259eab64a61c11.
> 2017-12-31 09:54:21,380 INFO [RS_OPEN_REGION-node1:16020-1]
> regionserver.HRegion: Onlined ba17cd27e2822b91193359a273684ac2; next
> sequenceid=22282
> 2017-12-31 09:54:21,386 INFO
> [PostOpenDeployTasks:ba17cd27e2822b91193359a273684ac2]
> regionserver.HRegionServer: Post open deploy tasks for
> TestTable,00000000000000000335544320,1505391191559.ba17cd27e2822b91193359a273684ac2.
> 2017-12-31 09:54:40,649 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> regionserver.RSRpcServices: Close 166b9c45d7724f72fd126adb4445d6ec without
> moving
> 2017-12-31 09:54:40,655 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> regionserver.RSRpcServices: Close ba17cd27e2822b91193359a273684ac2 without
> moving
> 2017-12-31 09:54:40,680 INFO
> [StoreCloserThread-TestTable,00000000000000000335544320,1505391191559.ba17cd27e2822b91193359a273684ac2.-1]
> regionserver.HStore: Closed info
> 2017-12-31 09:54:40,680 INFO
> [StoreCloserThread-TestTable,00000000000000000262144000,1505391191559.166b9c45d7724f72fd126adb4445d6ec.-1]
> regionserver.HStore: Closed info
> 2017-12-31 09:54:40,702 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> regionserver.RSRpcServices: Close 98025e37220dbc821f862a91d05a96ef without
> moving
> 2017-12-31 09:54:40,703 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> regionserver.RSRpcServices: Close 19d79b967c2e59cef288874533c154c8 without
> moving
> 2017-12-31 09:54:40,707 INFO
> [StoreCloserThread-TestTable,00000000000000000482344960,1505391191559.98025e37220dbc821f862a91d05a96ef.-1]
> regionserver.HStore: Closed info
> 2017-12-31 09:54:40,741 INFO [RS_CLOSE_REGION-node1:16020-0]
> regionserver.HRegion: Closed
> TestTable,00000000000000000262144000,1505391191559.166b9c45d7724f72fd126adb4445d6ec.
> 2017-12-31 09:54:40,754 INFO
> [StoreCloserThread-TestTable,00000000000000000555745280,1505391191559.19d79b967c2e59cef288874533c154c8.-1]
> regionserver.HStore: Closed info
> 2017-12-31 09:54:40,762 INFO [RS_CLOSE_REGION-node1:16020-1]
> regionserver.HRegion: Closed
> TestTable,00000000000000000335544320,1505391191559.ba17cd27e2822b91193359a273684ac2.
> 2017-12-31 09:54:40,807 INFO [RS_CLOSE_REGION-node1:16020-2]
> regionserver.HRegion: Closed
> TestTable,00000000000000000482344960,1505391191559.98025e37220dbc821f862a91d05a96ef.
> 2017-12-31 09:54:40,839 INFO [RS_CLOSE_REGION-node1:16020-0]
> regionserver.HRegion: Closed
> TestTable,00000000000000000555745280,1505391191559.19d79b967c2e59cef288874533c154c8.
> 2017-12-31 09:54:41,102 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> regionserver.RSRpcServices: Close f49f3cbb7f3db4cf96c7eb3b0cf83869 without
> moving
> 2017-12-31 09:54:41,105 INFO
> [StoreCloserThread-TestTable,00000000000000000408944640,1505391191559.f49f3cbb7f3db4cf96c7eb3b0cf83869.-1]
> regionserver.HStore: Closed info
> 2017-12-31 09:54:41,184 INFO [RS_CLOSE_REGION-node1:16020-1]
> regionserver.HRegion: Closed
> TestTable,00000000000000000408944640,1505391191559.f49f3cbb7f3db4cf96c7eb3b0cf83869.
> 2017-12-31 09:54:42,249 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> regionserver.RSRpcServices: Close e6ad456daff7a3d4ad259eab64a61c11 without
> moving
> 2017-12-31 09:54:42,250 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> regionserver.RSRpcServices: Close 330f09f4a0eaf26811c320fbf1b14e70 without
> moving
> 2017-12-31 09:54:42,250 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> regionserver.RSRpcServices: Close 655b0a0608ebaecacfda2e25a7e4b0f5 without
> moving
> 2017-12-31 09:54:42,253 INFO
> [StoreCloserThread-TestTable,00000000000000000115343360,1505391191559.e6ad456daff7a3d4ad259eab64a61c11.-1]
> regionserver.HStore: Closed info
> 2017-12-31 09:54:42,254 INFO
> [StoreCloserThread-TestTable,00000000000000000188743680,1505391191559.330f09f4a0eaf26811c320fbf1b14e70.-1]
> regionserver.HStore: Closed info
> 2017-12-31 09:54:42,254 INFO
> [StoreCloserThread-TestTable,00000000000000000178257920,1505391191559.655b0a0608ebaecacfda2e25a7e4b0f5.-1]
> regionserver.HStore: Closed info
> 2017-12-31 09:54:42,341 INFO [RS_CLOSE_REGION-node1:16020-1]
> regionserver.HRegion: Closed
> TestTable,00000000000000000178257920,1505391191559.655b0a0608ebaecacfda2e25a7e4b0f5.
> 2017-12-31 09:54:42,360 INFO [RS_CLOSE_REGION-node1:16020-2]
> regionserver.HRegion: Closed
> TestTable,00000000000000000115343360,1505391191559.e6ad456daff7a3d4ad259eab64a61c11.
> 2017-12-31 09:54:42,361 INFO [RS_CLOSE_REGION-node1:16020-0]
> regionserver.HRegion: Closed
> TestTable,00000000000000000188743680,1505391191559.330f09f4a0eaf26811c320fbf1b14e70.
> 2017-12-31 09:54:42,412 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> regionserver.RSRpcServices: Close 30eaa72ddc1f2706dc84c62e3222a006 without
> moving
> 2017-12-31 09:54:42,415 INFO
> [StoreCloserThread-TestTable,00000000000000000041943040,1505391191559.30eaa72ddc1f2706dc84c62e3222a006.-1]
> regionserver.HStore: Closed info
> 2017-12-31 09:54:42,495 INFO [RS_CLOSE_REGION-node1:16020-1]
> regionserver.HRegion: Closed
> TestTable,00000000000000000041943040,1505391191559.30eaa72ddc1f2706dc84c62e3222a006.
> 2017-12-31 09:54:53,069 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient:
> 0x1df82317 no activities for 60000 ms, close active connection. Will
> reconnect next time when there are new requests.
> 2017-12-31 09:54:53,088 INFO [ReadOnlyZKClient] zookeeper.ZooKeeper:
> Session: 0x160a94d45ab0122 closed
> 2017-12-31 09:54:53,088 INFO [ReadOnlyZKClient-EventThread]
> zookeeper.ClientCnxn: EventThread shut down for session: 0x160a94d45ab0122
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
