[
https://issues.apache.org/jira/browse/PHOENIX-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516234#comment-14516234
]
Dmitry Goldenberg commented on PHOENIX-1926:
--------------------------------------------
Josh,
Thanks for your help. It looks like I got this to work, with the following in
my spark-submit invocation:
{code}
./bin/spark-submit \
--driver-class-path $HBASE_HOME/lib/hbase-protocol-0.98.9-hadoop2.jar \
--driver-java-options
"-Dspark.driver.extraClassPath=$HBASE_HOME/lib/hbase-protocol-0.98.9-hadoop2.jar"
\
--class "com.myco.Driver" \
--master local[*] \
/mnt/data/myco.jar \
{code}
The crucial part to getting this to work was the first parameter,
--driver-class-path. Things work with just that and without
spark.driver.extraClassPath. Things do not work with
spark.driver.extraClassPath but no --driver-class-path, and of course they
don't work with both of these missing in the invocation.
I also have Phoenix' dependency hbase and hadoop jars' classes rolled into my
driver jar. Tested with them in the job jar and without, the error goes away in
either case if the protocol jar is on the driver-class-path.
Since I've only tried this in the local mode, it's not yet clear to me whether
spark.driver.extraClassPath and/or rolling the Phoenix' dependency hbase and
hadoop jars into the job jar would be required. But the good news is that
there's clearly a 'mojo' for getting this to work so one gets no
IllegalAccessError anymore.
Thanks again for your help.
> Attempt to update a record in HBase via Phoenix from a Spark job causes
> java.lang.IllegalAccessError: class
> com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass
> com.google.protobuf.LiteralByteString
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-1926
> URL: https://issues.apache.org/jira/browse/PHOENIX-1926
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.3.1
> Environment: centos x86_64 GNU/Linux
> Phoenix: 4.3.1 custom built for HBase 0.98.9, Hadoop 2.4.0
> HBase: 0.98.9-hadoop2
> Hadoop: 2.4.0
> Spark: spark-1.3.0-bin-hadoop2.4
> Reporter: Dmitry Goldenberg
> Priority: Critical
>
> Performing an UPSERT from within a Spark job,
> UPSERT INTO items(entry_id, prop1, pop2) VALUES(?, ?, ?)
> causes
> 15/04/26 18:22:06 WARN client.HTable: Error calling coprocessor service
> org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService for
> row \x00\x00ITEMS
> java.util.concurrent.ExecutionException: java.lang.IllegalAccessError: class
> com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass
> com.google.protobuf.LiteralByteString
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1620)
> at
> org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1577)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1007)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1257)
> at
> org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:350)
> at
> org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:311)
> at
> org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:307)
> at
> org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:333)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:237)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:231)
> at
> org.apache.phoenix.compile.FromCompiler.getResolverForMutation(FromCompiler.java:207)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:248)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:503)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:494)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:295)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:288)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:287)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:219)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:174)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:179)
> ...........................................
> Caused by: java.lang.IllegalAccessError: class
> com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass
> com.google.protobuf.LiteralByteString
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$7.call(ConnectionQueryServicesImpl.java:1265)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$7.call(ConnectionQueryServicesImpl.java:1258)
> at org.apache.hadoop.hbase.client.HTable$17.call(HTable.java:1608)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
