Josh Elser created HBASE-19769:
----------------------------------
Summary: IllegalAccessError on package-private Hadoop metrics2
classes in MapReduce jobs
Key: HBASE-19769
URL: https://issues.apache.org/jira/browse/HBASE-19769
Project: HBase
Issue Type: Bug
Components: mapreduce, metrics
Affects Versions: 2.0.0-beta-1
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Critical
Fix For: 2.0.0-beta-2
issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has been
using a custom classloader to separate Hadoop dependencies from the user's JAR
being run. A separate classloader is created for the user-provided jar, and
then this classloader is set as the contextClassLoader before the Tool is
executed by Hadoop's RunJar class. This has been (mostly?) fine for us to date
because we don't try to access any Hadoop internal classes client-side.
However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
problem is that Hadoop metrics2 implementations which we reference from the
same package are loaded by a different classloader than our HBase code is
loaded from. This makes the expected package-private access of these Metrics2
classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
{noformat}
java.lang.RuntimeException: Could not create interface
org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
compatibility jar on the classpath?
at
org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
at org.apache.hadoop.hbase.zookeeper.ZKMetrics.<init>(ZKMetrics.java:36)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:115)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
at
org.apache.hadoop.hbase.zookeeper.ZKWatcher.<init>(ZKWatcher.java:128)
at
org.apache.hadoop.hbase.zookeeper.ZKWatcher.<init>(ZKWatcher.java:102)
at
org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
at
org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
at
org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
at
org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
at
org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
at
org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
at
org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at
org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.util.ServiceConfigurationError:
org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at
java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at
org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
... 21 more
Caused by: java.lang.IllegalAccessError: tried to access class
org.apache.hadoop.metrics2.lib.MetricsInfoImpl from class
org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry
at
org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry.newGauge(DynamicMetricsRegistry.java:139)
at
org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl.<init>(MetricsZooKeeperSourceImpl.java:59)
at
org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl.<init>(MetricsZooKeeperSourceImpl.java:51)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at
java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 24 more
{noformat}
There are two ways to fix this problem that I see now:
# Remove client-side ZK metrics and disallow any more client-side metrics that
directly use metrics2 API
# Use public API for metrics2 only in HBase, or copy implementations from
Hadoop to avoid the need to access the package-private classes from this goofy
state.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
