[
https://issues.apache.org/jira/browse/PHOENIX-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234665#comment-14234665
]
James Taylor commented on PHOENIX-1473:
---------------------------------------
Ok, [~mujtaba], please go ahead with our RC.
We can't work around this as we were able to in 0.94, as the RS is aborting now
due to our coprocessor not being found. Other than telling the user to "look
through the server logs", there's nothing we can do, as our client just hangs
(for 35 retries which seems crazy too) and then just comes back with the
exception that the connection was refused. The only way around this is to set
the following in the hbase-sites.xml:
<property>
<name>hbase.coprocessor.abortonerror</name>
<value>false</value>
</property>
But it really makes no sense to recommend this because the user might as well
correctly put the phoenix.jar in the lib directory instead.
Hopefully we'll get a fix for HBASE-12570 soon, but otherwise our users just
have to do the install correctly at the risk of bringing down their entire
cluster (which is pretty good incentive to get it right...)
> Connecting with Phoenix client when Phoenix is not deployed on region
> server(s) takes down region server(s).
> ------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-1473
> URL: https://issues.apache.org/jira/browse/PHOENIX-1473
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.2
> Reporter: Serhiy Bilousov
> Assignee: James Taylor
> Fix For: 4.3, 4.2.2
>
> Attachments: PHOENIX-1473.patch, PHOENIX-1473_4_2.patch
>
>
> When attempting to connect with Phoenix client when Phoenix server not
> deployed will take down region servers. It looks like problem with creating
> SYSTEM.CATALOG table causing it. It may be argued that there some
> improvements can be done on HBase side in regards to not allow to create
> table if coprocessor jar files could not be found but from Phoenix side I
> would think before doing anything it should check if required jar is in place
> (meaning Phoenix deployed properly).
> here is log from region server
> 2014-11-20 14:29:48,411 ERROR [RS_OPEN_REGION-dn01:60020-1]
> handler.OpenRegionHandler: Failed open of
> region=SYSTEM.CATALOG,,1416493787900.35b7b3a19f75688fce382e8f1323b4ae.,
> starting to roll back the global memstore size.
> java.io.IOException: Unable to load configured region split policy
> 'org.apache.phoenix.schema.MetaDataSplitPolicy' for table 'SYSTEM.CATALOG'
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPolicyClass(RegionSplitPolicy.java:121)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.create(RegionSplitPolicy.java:101)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:727)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:684)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4550)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4520)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4492)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4448)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4399)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.phoenix.schema.MetaDataSplitPolicy
> at java.net.URLClassLoader$1.run(Unknown Source)
> at java.net.URLClassLoader$1.run(Unknown Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(Unknown Source)
> at java.lang.ClassLoader.loadClass(Unknown Source)
> at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
> at java.lang.ClassLoader.loadClass(Unknown Source)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Unknown Source)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPolicyClass(RegionSplitPolicy.java:117)
> ... 14 more
> 2014-11-20 14:29:48,411 INFO [RS_OPEN_REGION-dn01:60020-1]
> handler.OpenRegionHandler: Opening of region {ENCODED =>
> 35b7b3a19f75688fce382e8f1323b4ae, NAME =>
> 'SYSTEM.CATALOG,,1416493787900.35b7b3a19f75688fce382e8f1323b4ae.', STARTKEY
> => '', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK,
> expecting version 4
> 2014-11-20 14:29:48,412 DEBUG [RS_OPEN_REGION-dn01:60020-1]
> zookeeper.ZKAssign: regionserver:60020-0x349cd39697a00e5,
> quorum=nn02.abc.com:2181,nn01.abc.com:2181,jz01.abc.com:2181,
> baseZNode=/hbase Transitioning 35b7b3a19f75688fce382e8f1323b4ae from
> RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 2014-11-20 14:29:48,414 DEBUG [RS_OPEN_REGION-dn01:60020-1]
> zookeeper.ZKAssign: regionserver:60020-0x349cd39697a00e5,
> quorum=nn02.abc.com:2181,nn01.abc.com:2181,z01.abc.com:2181, baseZNode=/hbase
> Transitioned node 35b7b3a19f75688fce382e8f1323b4ae from RS_ZK_REGION_OPENING
> to RS_ZK_REGION_FAILED_OPEN
> 2014-11-20 14:29:48,528 INFO [regionserver60020] regionserver.HRegionServer:
> stopping server dn01.abc.com,60020,1416493546924; all regions closed.
> 2014-11-20 14:29:48,528 DEBUG [regionserver60020-WAL.AsyncNotifier]
> wal.FSHLog: regionserver60020-WAL.AsyncNotifier interrupted while waiting for
> notification from AsyncSyncer thread
> 2014-11-20 14:29:48,528 INFO [regionserver60020-WAL.AsyncNotifier]
> wal.FSHLog: regionserver60020-WAL.AsyncNotifier exiting
> 2014-11-20 14:29:48,529 DEBUG [regionserver60020-WAL.AsyncSyncer0]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer0 interrupted while waiting for
> notification from AsyncWriter thread
> 2014-11-20 14:29:48,529 INFO [regionserver60020-WAL.AsyncSyncer0]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer0 exiting
> 2014-11-20 14:29:48,529 DEBUG [regionserver60020-WAL.AsyncSyncer1]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer1 interrupted while waiting for
> notification from AsyncWriter thread
> 2014-11-20 14:29:48,529 INFO [regionserver60020-WAL.AsyncSyncer1]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer1 exiting
> 2014-11-20 14:29:48,529 DEBUG [regionserver60020-WAL.AsyncSyncer2]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer2 interrupted while waiting for
> notification from AsyncWriter thread
> 2014-11-20 14:29:48,529 INFO [regionserver60020-WAL.AsyncSyncer2]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer2 exiting
> 2014-11-20 14:29:48,529 DEBUG [regionserver60020-WAL.AsyncSyncer3]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer3 interrupted while waiting for
> notification from AsyncWriter thread
> 2014-11-20 14:29:48,529 INFO [regionserver60020-WAL.AsyncSyncer3]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer3 exiting
> 2014-11-20 14:29:48,530 DEBUG [regionserver60020-WAL.AsyncSyncer4]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer4 interrupted while waiting for
> notification from AsyncWriter thread
> 2014-11-20 14:29:48,530 INFO [regionserver60020-WAL.AsyncSyncer4]
> wal.FSHLog: regionserver60020-WAL.AsyncSyncer4 exiting
> 2014-11-20 14:29:48,530 DEBUG [regionserver60020-WAL.AsyncWriter] wal.FSHLog:
> regionserver60020-WAL.AsyncWriter interrupted while waiting for newer writes
> added to local buffer
> 2014-11-20 14:29:48,530 INFO [regionserver60020-WAL.AsyncWriter] wal.FSHLog:
> regionserver60020-WAL.AsyncWriter exiting
> 2014-11-20 14:29:48,530 DEBUG [regionserver60020] wal.FSHLog: Closing WAL
> writer in hdfs://mycluster/hbase/WALs/dn01.abc.com,60020,1416493546924
> 2014-11-20 14:29:48,547 INFO [regionserver60020] regionserver.Leases:
> regionserver60020 closing leases
> 2014-11-20 14:29:48,547 INFO [regionserver60020] regionserver.Leases:
> regionserver60020 closed leases
> 2014-11-20 14:29:48,915 INFO [regionserver60020.periodicFlusher]
> regionserver.HRegionServer$PeriodicMemstoreFlusher:
> regionserver60020.periodicFlusher exiting
> 2014-11-20 14:29:48,915 INFO [regionserver60020]
> regionserver.CompactSplitThread: Waiting for Split Thread to finish...
> 2014-11-20 14:29:48,915 INFO [regionserver60020]
> regionserver.CompactSplitThread: Waiting for Merge Thread to finish...
> 2014-11-20 14:29:48,915 INFO [regionserver60020]
> regionserver.CompactSplitThread: Waiting for Large Compaction Thread to
> finish...
> 2014-11-20 14:29:48,916 INFO [regionserver60020]
> regionserver.CompactSplitThread: Waiting for Small Compaction Thread to
> finish...
> 2014-11-20 14:29:48,920 INFO [regionserver60020]
> client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x349cd39697a00e6
> 2014-11-20 14:29:48,922 INFO [regionserver60020-EventThread]
> zookeeper.ClientCnxn: EventThread shut down
> 2014-11-20 14:29:48,922 INFO [regionserver60020] zookeeper.ZooKeeper:
> Session: 0x349cd39697a00e6 closed
> 2014-11-20 14:29:48,922 INFO [regionserver60020.leaseChecker]
> regionserver.Leases: regionserver60020.leaseChecker closing leases
> 2014-11-20 14:29:48,922 INFO [regionserver60020.leaseChecker]
> regionserver.Leases: regionserver60020.leaseChecker closed leases
> 2014-11-20 14:29:48,926 INFO [regionserver60020-EventThread]
> zookeeper.ClientCnxn: EventThread shut down
> 2014-11-20 14:29:48,926 INFO [regionserver60020] zookeeper.ZooKeeper:
> Session: 0x349cd39697a00e5 closed
> 2014-11-20 14:29:48,926 INFO [regionserver60020] regionserver.HRegionServer:
> stopping server dn01.abc.com,60020,1416493546924; zookeeper connection closed.
> 2014-11-20 14:29:48,926 INFO [regionserver60020] regionserver.HRegionServer:
> regionserver60020 exiting
> 2014-11-20 14:29:48,926 ERROR [main] regionserver.HRegionServerCommandLine:
> Region server exiting
> java.lang.RuntimeException: HRegionServer Aborted
> at
> org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2422)
> 2014-11-20 14:29:48,928 INFO [Thread-9] regionserver.ShutdownHook: Shutdown
> hook starting; hbase.shutdown.hook=true;
> fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@3e44f2a5
> 2014-11-20 14:29:48,928 INFO [Thread-9] regionserver.ShutdownHook: Starting
> fs shutdown hook thread.
> 2014-11-20 14:29:48,930 INFO [Thread-9] regionserver.ShutdownHook: Shutdown
> hook finished.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
