More info...
By trial and error, I tested different EMR versions and made a little
incomplete list of which ones support Pig Phoenix storage and which ones
don't:
emr-5.8.0 JacksonJaxbJsonProvider error
emr-5.6.0 JacksonJaxbJsonProvider error
emr-5.4.0 JacksonJaxbJsonProvider error
emr-5.3.1 ContainerId.fromString() error
emr-5.3.0 ContainerId.fromString() error
emr-5.0.0 ContainerId.fromString() error
emr-4.9.2 Works!
emr-4.7.0 Works!
I ran out of time trying to get 5.8.0 working, so will start using 4.9.2.
But I would like to switch to 5.8.0 if anyone has a solution. Meanwhile, I
hope this list saves other people some time and headache.
Thanks,
Steve
On Thu, Aug 17, 2017 at 2:40 PM, Steve Terrell <sterr...@oculus360.us>
wrote:
> I'm running EMR 5.8.0 with these applications installed:
> Pig 0.16.0, Phoenix 4.11.0, HBase 1.3.1
>
> Here is my pig script (try.pig):
>
> REGISTER /usr/lib/phoenix/phoenix-4.11.0-HBase-1.3-client.jar;
> A = load '/steve/a.txt' as (TXT:chararray);
> store A into 'hbase://A_TABLE' using org.apache.phoenix.pig.
> PhoenixHBaseStorage('10.0.100.51','-batchSize 2500');
>
> I run it like this from the command line:
> pig try.pig
>
> When it fails, I dig into the hadoop task logs and find this:
> 2017-08-17 19:11:37,539 [ERROR] [main] |app.DAGAppMaster|: Error starting
> DAGAppMaster
> java.lang.NoClassDefFoundError: org/apache/phoenix/shaded/org/
> codehaus/jackson/jaxrs/JacksonJaxbJsonProvider
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceInit(
> TimelineClientImpl.java:269)
> at org.apache.hadoop.service.AbstractService.init(
> AbstractService.java:163)
> at org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService.
> serviceInit(ATSHistoryLoggingService.java:102)
> at org.apache.hadoop.service.AbstractService.init(
> AbstractService.java:163)
> at org.apache.hadoop.service.CompositeService.serviceInit(
> CompositeService.java:107)
> at org.apache.tez.dag.history.HistoryEventHandler.serviceInit(
> HistoryEventHandler.java:73)
> at org.apache.hadoop.service.AbstractService.init(
> AbstractService.java:163)
> at org.apache.tez.dag.app.DAGAppMaster.initServices(
> DAGAppMaster.java:1922)
> at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:624)
> at org.apache.hadoop.service.AbstractService.init(
> AbstractService.java:163)
> at org.apache.tez.dag.app.DAGAppMaster$8.run(DAGAppMaster.java:2557)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1698)
> at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(
> DAGAppMaster.java:2554)
> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2359)
> Caused by: java.lang.ClassNotFoundException: org.apache.phoenix.shaded.org
> .codehaus.jackson.jaxrs.JacksonJaxbJsonProvider
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 28 more
>
> Has anyone been able to get org.apache.phoenix.pig.PhoenixHBaseStorage()
> to work on recent EMR versions? Please help if you can.
>
> Thank you,
> Steve
>
