[
https://issues.apache.org/jira/browse/HBASE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871482#comment-13871482
]
Andrew Purtell commented on HBASE-6873:
---------------------------------------
Yeah but at least here someone is following along.
bq. That doesn't quite work for coprocessors configured as table attributes
(more like wipe the table dir from HDFS),
I was just looking for an issue I filed a while back, an option for pausing the
master after the deployment of system tables but before deployment of user
tables, but can't find it offhand. If meta is online it should be possible to
alter the HTD/HCDs.
> Clean up Coprocessor loading failure handling
> ---------------------------------------------
>
> Key: HBASE-6873
> URL: https://issues.apache.org/jira/browse/HBASE-6873
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors, regionserver
> Affects Versions: 0.98.0
> Reporter: David Arthur
> Assignee: Andrew Purtell
> Priority: Blocker
> Fix For: 0.98.0, 0.99.0
>
> Attachments: 6873.patch, 6873.patch, 6873.patch, 6873.patch,
> 6873.patch, 6873.patch
>
>
> When registering a coprocessor with a missing dependency, the regionserver
> gets stuck in an infinite fail loop. Restarting the regionserver and/or
> master has no affect.
> E.g.,
> Load coprocessor from my-coproc.jar, that uses an external dependency (kafka)
> that is not included with HBase.
> {code}
> 12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME =>
> 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '',
> ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking
> as FAILED_OPEN in ZK
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Attempting to transition node
> 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to
> RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Successfully transitioned node
> 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to
> RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open
> region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Attempting to transition node
> 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Successfully transitioned node
> 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME =>
> 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '',
> ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor
> config now ...
> 12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class
> com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded
> from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of
> region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting
> to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError:
> kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException:
> kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> 12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME =>
> 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '',
> ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking
> as FAILED_OPEN in ZK
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Attempting to transition node
> 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to
> RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Successfully transitioned node
> 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to
> RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open
> region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Attempting to transition node
> 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign:
> regionserver:60020-0x139f43af2a70043 Successfully transitioned node
> 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME =>
> 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '',
> ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor
> config now ...
> 12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class
> com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded
> from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of
> region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting
> to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError:
> kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException:
> kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> {code}
> Ad infinitum.
> It seems that upon failing to open a region after adding a coprocessor, that
> coprocessor should be unregister or at least disabled/blacklisted.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
