[ 
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383348#comment-16383348
 ] 

Hudson commented on HBASE-19769:
--------------------------------

Results for branch branch-1, done in 47 min and counting
        [build #236 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/236/]:
----
details (if available):

(x) *{color:red}-1 overall{color}*
    Committer, please check your recent inclusion of a patch for this issue.









(x) {color:red}-1 source release artifact{color}
-- See build output for details.


> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce 
> jobs
> -------------------------------------------------------------------------------
>
>                 Key: HBASE-19769
>                 URL: https://issues.apache.org/jira/browse/HBASE-19769
>             Project: HBase
>          Issue Type: Bug
>          Components: mapreduce, metrics
>    Affects Versions: 2.0.0-beta-1
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>            Priority: Critical
>             Fix For: 1.5.0, 2.0.0-beta-2, 1.4.3
>
>         Attachments: HBASE-19769.001.branch-2.patch
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has 
> been using a custom classloader to separate Hadoop dependencies from the 
> user's JAR being run. A separate classloader is created for the user-provided 
> jar, and then this classloader is set as the contextClassLoader before the 
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for 
> us to date because we don't try to access any Hadoop internal classes 
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The 
> problem is that Hadoop metrics2 implementations which we reference from the 
> same package are loaded by a different classloader than our HBase code is 
> loaded from. This makes the expected package-private access of these Metrics2 
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create  interface 
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop 
> compatibility jar on the classpath?
>         at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
>         at 
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.<init>(ZKMetrics.java:36)
>         at 
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:115)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
>         at 
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.<init>(ZKWatcher.java:128)
>         at 
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.<init>(ZKWatcher.java:102)
>         at 
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
>         at 
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
>         at 
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
>         at 
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
>         at 
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
>         at 
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
>         at 
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>         at 
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError: 
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider 
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be 
> instantiated
>         at java.util.ServiceLoader.fail(ServiceLoader.java:232)
>         at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
>         at 
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
>         at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>         at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>         at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
>         ... 21 more
> Caused by: java.lang.IllegalAccessError: tried to access class 
> org.apache.hadoop.metrics2.lib.MetricsInfoImpl from class 
> org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry
>         at 
> org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry.newGauge(DynamicMetricsRegistry.java:139)
>         at 
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl.<init>(MetricsZooKeeperSourceImpl.java:59)
>         at 
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl.<init>(MetricsZooKeeperSourceImpl.java:51)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>         at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>         at java.lang.Class.newInstance(Class.java:442)
>         at 
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
>         ... 24 more
> {noformat}
> There are two ways to fix this problem that I see now:
> # Remove client-side ZK metrics and disallow any more client-side metrics 
> that directly use metrics2 API
> # Use public API for metrics2 only in HBase, or copy implementations from 
> Hadoop to avoid the need to access the package-private classes from this 
> goofy state.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to