[
https://issues.apache.org/jira/browse/HBASE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-6873:
----------------------------------
Release Note: The default setting of the configuration parameter
'hbase.coprocessor.abortonerror' has been changed from "false" to "true". This
will cause HBase processes to abort if a coprocessor cannot be loaded, fails to
initialize, or throws an unexpected exception. If "false", this instead will
allow the server to continue execution in such cases but the system wide state
of the coprocessor in question will become inconsistent as it will be properly
executing in only a subset of servers, so this is most useful for debugging a
new coprocessor deployment on a test cluster. Running a secure production
cluster with this setting set to "false" is not advised. (was: The default
setting of the configuration parameter 'hbase.coprocessor.abortonerror' has
been changed from "false" to "true". This will cause HBase processes to abort
if a coprocessor cannot be loaded, fails to initialize, or throws an unexpected
exception. If "false", this instead will allow the server to continue execution
in such cases but the system wide state of the coprocessor in question will
become inconsistent as it will be properly executing in only a subset of
servers, so this is most useful for debugging only.)
> Clean up Coprocessor loading failure handling
> ---------------------------------------------
>
> Key: HBASE-6873
> URL: https://issues.apache.org/jira/browse/HBASE-6873
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors, regionserver
> Affects Versions: 0.98.0
> Reporter: David Arthur
> Assignee: Andrew Purtell
> Priority: Blocker
> Fix For: 0.98.0, 0.99.0
>
> Attachments: 6873.patch, 6873.patch, 6873.patch, 6873.patch,
> 6873.patch, 6873.patch, 6873.patch
>
>
> When registering a coprocessor with a missing dependency, the regionserver
> gets stuck in an infinite fail loop. Restarting the regionserver and/or
> master has no affect.
> E.g.,
> Load coprocessor from my-coproc.jar, that uses an external dependency (kafka)
> that is not included with HBase.
> {code}
> 12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME =>
> 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '',
> ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking
> as FAILED_OPEN in ZK
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Attempting to transition node
> 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to
> RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Successfully transitioned node
> 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to
> RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open
> region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Attempting to transition node
> 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Successfully transitioned node
> 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME =>
> 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '',
> ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor
> config now ...
> 12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class
> com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded
> from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of
> region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting
> to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError:
> kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException:
> kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> 12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME =>
> 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '',
> ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking
> as FAILED_OPEN in ZK
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Attempting to transition node
> 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to
> RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Successfully transitioned node
> 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to
> RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open
> region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Attempting to transition node
> 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Successfully transitioned node
> 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME =>
> 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '',
> ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor
> config now ...
> 12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class
> com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded
> from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of
> region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting
> to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError:
> kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException:
> kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> {code}
> Ad infinitum.
> It seems that upon failing to open a region after adding a coprocessor, that
> coprocessor should be unregister or at least disabled/blacklisted.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
