[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19769:
-------------------------------
Attachment: HBASE-19769.001.branch-2.patch
.001 Parking an initial patch that removes the ZKMetrics.
Will put this on my test cluster to verify it works.
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> -------------------------------------------------------------------------------
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
> Affects Versions: 2.0.0-beta-1
> Reporter: Josh Elser
> Assignee: Josh Elser
> Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19769.001.branch-2.patch
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.<init>(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.<init>(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.<init>(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
> ... 21 more
> Caused by: java.lang.IllegalAccessError: tried to access class
> org.apache.hadoop.metrics2.lib.MetricsInfoImpl from class
> org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry
> at
> org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry.newGauge(DynamicMetricsRegistry.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl.<init>(MetricsZooKeeperSourceImpl.java:59)
> at
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl.<init>(MetricsZooKeeperSourceImpl.java:51)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 24 more
> {noformat}
> There are two ways to fix this problem that I see now:
> # Remove client-side ZK metrics and disallow any more client-side metrics
> that directly use metrics2 API
> # Use public API for metrics2 only in HBase, or copy implementations from
> Hadoop to avoid the need to access the package-private classes from this
> goofy state.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
