[
https://issues.apache.org/jira/browse/HBASE-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13868472#comment-13868472
]
Jimmy Xiang commented on HBASE-10304:
-------------------------------------
I tried with -libjars, and it gave me the same problem. So it is not working
for me.
I also tried the three suggestions. The first two of them need some tweaking,
while the third one work as-is.
bq. $ HADOOP_CLASSPATH=/path/to/hbase-protocol.jar hadoop jar MyJob.jar
MyJobMainClass
I got this:
{noformat}
14/01/10 15:31:05 WARN zookeeper.ClientCnxn: Session 0x0 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
{noformat}
Basically, I can't connect to the ZK. I have to add the hbase conf dir as
below:
{noformat}
$ HADOOP_CLASSPATH=/path/to/hbase-protocol.jar:/path/to/hbase-conf hadoop jar
MyJob.jar MyJobMainClass
{noformat}
bq. $ HADOOP_CLASSPATH=$(hbase mapredcp) hadoop jar MyJob.jar MyJobMainClass
Same as above. I need to add hbase conf dir to the path:
{noformat}
$ HADOOP_CLASSPATH=$(hbase mapredcp):/path/to/hbase-conf hadoop jar MyJob.jar
MyJobMainClass
{noformat}
bq. $ HADOOP_CLASSPATH=$(hbase classpath) hadoop jar MyJob.jar MyJobMainClass
Works for me.
> Running an hbase job jar: IllegalAccessError: class
> com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass
> com.google.protobuf.LiteralByteString
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-10304
> URL: https://issues.apache.org/jira/browse/HBASE-10304
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.98.0, 0.96.1.1
> Reporter: stack
> Priority: Blocker
> Fix For: 0.98.0
>
> Attachments: hbase-10304_not_tested.patch, jobjar.xml
>
>
> (Jimmy has been working on this one internally. I'm just the messenger
> raising this critical issue upstream).
> So, if you make job jar and bundle up hbase inside in it because you want to
> access hbase from your mapreduce task, the deploy of the job jar to the
> cluster fails with:
> {code}
> 14/01/05 08:59:19 INFO Configuration.deprecation:
> topology.node.switch.mapping.impl is deprecated. Instead, use
> net.topology.node.switch.mapping.impl
> 14/01/05 08:59:19 INFO Configuration.deprecation: io.bytes.per.checksum is
> deprecated. Instead, use dfs.bytes-per-checksum
> Exception in thread "main" java.lang.IllegalAccessError: class
> com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass
> com.google.protobuf.LiteralByteString
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:792)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:818)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.convertScanToString(TableMapReduceUtil.java:433)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:186)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:147)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:270)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:100)
> at
> com.ngdata.hbaseindexer.mr.HBaseMapReduceIndexerTool.run(HBaseMapReduceIndexerTool.java:124)
> at
> com.ngdata.hbaseindexer.mr.HBaseMapReduceIndexerTool.run(HBaseMapReduceIndexerTool.java:64)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> com.ngdata.hbaseindexer.mr.HBaseMapReduceIndexerTool.main(HBaseMapReduceIndexerTool.java:51)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
> So, ZCLBS is a hack. This class is in the hbase-protocol module. It is "in"
> the com.google.protobuf package. All is well and good usually.
> But when we make a job jar and bundle up hbase inside it, our 'trick' breaks.
> RunJar makes a new class loader to run the job jar. This URLCLassLoader
> 'attaches' all the jars and classes that are in jobjar so they can be found
> when it does to do a lookup only Classloaders work by always delegating to
> their parent first (unless you are a WAR file in a container where delegation
> is 'off' for the most part) and in this case, the parent classloader will
> have access to a pb jar since pb is in the hadoop CLASSPATH. So, the parent
> loads the pb classes.
> We then load ZCLBS only this is done in the claslsloader made by RunJar;
> ZKCLBS has a different classloader from its superclass and we get the above
> IllegalAccessError.
> Now (Jimmy's work comes in here), this can't be fixed by reflection -- you
> can't setAccess on a 'Class' -- and though it probably could be fixed by
> hacking RunJar so it was somehow made configurable so we could put in place
> our own ClassLoader to do something like containers do for WAR files
> (probably not a bad idea), there would be some fierce hackery involved and
> besides, this won't show up in hadoop anytime too soon leaving hadoop 2.2ers
> out in the cold.
> So, the alternatives are:
> 1. Undo the ZCLSB hack. We'd lose a lot of nice perf improvement but I'd say
> this is preferable to crazy CLASSPATH hacks.
> 2. Require folks put hbase-protocol -- thats all you'd need -- on the hadoop
> CLASSPATH. This is kinda crazy.
> 3. We could try shading the pb jar content or probably better, just pull pb
> into hbase altogether only under a different package. If it was in our code
> base, we could do more ZCLSB-like speedups.
> I was going to experiment with #3 above unless anyone else has a better idea.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
