[
https://issues.apache.org/jira/browse/PHOENIX-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219701#comment-14219701
]
Lars Hofhansl commented on PHOENIX-1473:
----------------------------------------
This is a bit of a slippery slope. If a table already exists and the split
policy cannot be loaded the region server has no choice but to shut itself down
- it can't know what was intended with the split policy and might mess things
up. Well, or it could disable the table.
For new tables that is different story as James (and Andrew) point out. Maybe
the answer there is as well to allow creating the table, but block enabling
it...?
> Connecting with Phoenix client when Phoenix is not deployed on region
> server(s) takes down region server(s).
> ------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-1473
> URL: https://issues.apache.org/jira/browse/PHOENIX-1473
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.2
> Reporter: SERHIY BILOUSOV
>
> When attempting to connect with Phoenix client when Phoenix server not
> deployed will take down region servers. It looks like problem with creating
> SYSTEM.CATALOG table causing it. It may be argued that there some
> improvements can be done on HBase side in regards to not allow to create
> table if coprocessor jar files could not be found but from Phoenix side I
> would think before doing anything it should check if required jar is in place
> (meaning Phoenix deployed properly).
> here is log from region server
> 2014-11-20 14:29:48,411 ERROR [RS_OPEN_REGION-dn01:60020-1]
> handler.OpenRegionHandler: Failed open of
> region=SYSTEM.CATALOG,,1416493787900.35b7b3a19f75688fce382e8f1323b4ae.,
> starting to roll back the global memstore size.
> java.io.IOException: Unable to load configured region split policy
> 'org.apache.phoenix.schema.MetaDataSplitPolicy' for table 'SYSTEM.CATALOG'
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPolicyClass(RegionSplitPolicy.java:121)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.create(RegionSplitPolicy.java:101)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:727)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:684)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4550)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4520)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4492)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4448)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4399)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.phoenix.schema.MetaDataSplitPolicy
> at java.net.URLClassLoader$1.run(Unknown Source)
> at java.net.URLClassLoader$1.run(Unknown Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(Unknown Source)
> at java.lang.ClassLoader.loadClass(Unknown Source)
> at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
> at java.lang.ClassLoader.loadClass(Unknown Source)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Unknown Source)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPolicyClass(RegionSplitPolicy.java:117)
> ... 14 more
> 2014-11-20 14:29:48,411 INFO [RS_OPEN_REGION-dn01:60020-1]
> handler.OpenRegionHandler: Opening of region {ENCODED =>
> 35b7b3a19f75688fce382e8f1323b4ae, NAME =>
> 'SYSTEM.CATALOG,,1416493787900.35b7b3a19f75688fce382e8f1323b4ae.', STARTKEY
> => '', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK,
> expecting version 4
> 2014-11-20 14:29:48,412 DEBUG [RS_OPEN_REGION-dn01:60020-1]
> zookeeper.ZKAssign: regionserver:60020-0x349cd39697a00e5,
> quorum=nn02.abc.com:2181,nn01.abc.com:2181,jz01.abc.com:2181,
> baseZNode=/hbase Transitioning 35b7b3a19f75688fce382e8f1323b4ae from
> RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 2014-11-20 14:29:48,414 DEBUG [RS_OPEN_REGION-dn01:60020-1]
> zookeeper.ZKAssign: regionserver:60020-0x349cd39697a00e5,
> quorum=nn02.abc.com:2181,nn01.abc.com:2181,z01.abc.com:2181, baseZNode=/hbase
> Transitioned node 35b7b3a19f75688fce382e8f1323b4ae from RS_ZK_REGION_OPENING
> to RS_ZK_REGION_FAILED_OPEN
> 2014-11-20 14:29:48,528 INFO [regionserver60020] regionserver.HRegionServer:
> stopping server dn01.abc.com,60020,1416493546924; all regions closed.
> 2014-11-20 14:29:48,528 DEBUG [regionserver60020-WAL.AsyncNotifier]
> wal.FSHLog: regionserver60020-WAL.AsyncNotifier interrupted while waiting for
> notification from AsyncSyncer thread
> 2014-11-20 14:29:48,528 INFO [regionserver60020-WAL.AsyncNotifier]
> wal.FSHLog: regionserver60020-WAL.AsyncNotifier exiting
> 2014-11-20 14:29:48,529 DEBUG [regionserver60020-WAL.AsyncSyncer0]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer0 interrupted while waiting for
> notification from AsyncWriter thread
> 2014-11-20 14:29:48,529 INFO [regionserver60020-WAL.AsyncSyncer0]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer0 exiting
> 2014-11-20 14:29:48,529 DEBUG [regionserver60020-WAL.AsyncSyncer1]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer1 interrupted while waiting for
> notification from AsyncWriter thread
> 2014-11-20 14:29:48,529 INFO [regionserver60020-WAL.AsyncSyncer1]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer1 exiting
> 2014-11-20 14:29:48,529 DEBUG [regionserver60020-WAL.AsyncSyncer2]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer2 interrupted while waiting for
> notification from AsyncWriter thread
> 2014-11-20 14:29:48,529 INFO [regionserver60020-WAL.AsyncSyncer2]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer2 exiting
> 2014-11-20 14:29:48,529 DEBUG [regionserver60020-WAL.AsyncSyncer3]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer3 interrupted while waiting for
> notification from AsyncWriter thread
> 2014-11-20 14:29:48,529 INFO [regionserver60020-WAL.AsyncSyncer3]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer3 exiting
> 2014-11-20 14:29:48,530 DEBUG [regionserver60020-WAL.AsyncSyncer4]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer4 interrupted while waiting for
> notification from AsyncWriter thread
> 2014-11-20 14:29:48,530 INFO [regionserver60020-WAL.AsyncSyncer4]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer4 exiting
> 2014-11-20 14:29:48,530 DEBUG [regionserver60020-WAL.AsyncWriter] wal.FSHLog:
> regionserver60020-WAL.AsyncWriter interrupted while waiting for newer writes
> added to local buffer
> 2014-11-20 14:29:48,530 INFO [regionserver60020-WAL.AsyncWriter] wal.FSHLog:
> regionserver60020-WAL.AsyncWriter exiting
> 2014-11-20 14:29:48,530 DEBUG [regionserver60020] wal.FSHLog: Closing WAL
> writer in hdfs://mycluster/hbase/WALs/dn01.abc.com,60020,1416493546924
> 2014-11-20 14:29:48,547 INFO [regionserver60020] regionserver.Leases:
> regionserver60020 closing leases
> 2014-11-20 14:29:48,547 INFO [regionserver60020] regionserver.Leases:
> regionserver60020 closed leases
> 2014-11-20 14:29:48,915 INFO [regionserver60020.periodicFlusher]
> regionserver.HRegionServer$PeriodicMemstoreFlusher:
> regionserver60020.periodicFlusher exiting
> 2014-11-20 14:29:48,915 INFO [regionserver60020]
> regionserver.CompactSplitThread: Waiting for Split Thread to finish...
> 2014-11-20 14:29:48,915 INFO [regionserver60020]
> regionserver.CompactSplitThread: Waiting for Merge Thread to finish...
> 2014-11-20 14:29:48,915 INFO [regionserver60020]
> regionserver.CompactSplitThread: Waiting for Large Compaction Thread to
> finish...
> 2014-11-20 14:29:48,916 INFO [regionserver60020]
> regionserver.CompactSplitThread: Waiting for Small Compaction Thread to
> finish...
> 2014-11-20 14:29:48,920 INFO [regionserver60020]
> client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x349cd39697a00e6
> 2014-11-20 14:29:48,922 INFO [regionserver60020-EventThread]
> zookeeper.ClientCnxn: EventThread shut down
> 2014-11-20 14:29:48,922 INFO [regionserver60020] zookeeper.ZooKeeper:
> Session: 0x349cd39697a00e6 closed
> 2014-11-20 14:29:48,922 INFO [regionserver60020.leaseChecker]
> regionserver.Leases: regionserver60020.leaseChecker closing leases
> 2014-11-20 14:29:48,922 INFO [regionserver60020.leaseChecker]
> regionserver.Leases: regionserver60020.leaseChecker closed leases
> 2014-11-20 14:29:48,926 INFO [regionserver60020-EventThread]
> zookeeper.ClientCnxn: EventThread shut down
> 2014-11-20 14:29:48,926 INFO [regionserver60020] zookeeper.ZooKeeper:
> Session: 0x349cd39697a00e5 closed
> 2014-11-20 14:29:48,926 INFO [regionserver60020] regionserver.HRegionServer:
> stopping server dn01.abc.com,60020,1416493546924; zookeeper connection closed.
> 2014-11-20 14:29:48,926 INFO [regionserver60020] regionserver.HRegionServer:
> regionserver60020 exiting
> 2014-11-20 14:29:48,926 ERROR [main] regionserver.HRegionServerCommandLine:
> Region server exiting
> java.lang.RuntimeException: HRegionServer Aborted
> at
> org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2422)
> 2014-11-20 14:29:48,928 INFO [Thread-9] regionserver.ShutdownHook: Shutdown
> hook starting; hbase.shutdown.hook=true;
> fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@3e44f2a5
> 2014-11-20 14:29:48,928 INFO [Thread-9] regionserver.ShutdownHook: Starting
> fs shutdown hook thread.
> 2014-11-20 14:29:48,930 INFO [Thread-9] regionserver.ShutdownHook: Shutdown
> hook finished.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
