And forgot to mention that we invoke our pig scripts through oozie.
On Mon, Aug 21, 2017 at 2:20 PM, anil gupta <anilgupt...@gmail.com> wrote:
> Sorry, cant share the pig script.
> Here is what we are registering:
> REGISTER /usr/lib/phoenix/phoenix-4.7.0-HBase-1.2-client.jar;
> REGISTER /usr/lib/pig/lib/piggybank.jar;
>
>
> Following is the classpath of Hadoop and Yarn:
> [hadoop@ip-52-143 ~]$ hadoop classpath
> /etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.
> //*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/
> usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/
> lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/
> lib/hadoop-mapreduce/.//*::/etc/tez/conf:/usr/lib/tez/*:/
> usr/lib/tez/lib/*:/usr/lib/hadoop-lzo/lib/*:/usr/share/
> aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/
> aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/
> usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/
> emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/
> emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/aws/emr/
> cloudwatch-sink/lib/*:/usr/share/aws/emr/security/conf:/
> usr/share/aws/emr/security/lib/*
> [hadoop@ip-52-143 ~]$ yarn classpath
> /etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/
> lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-
> hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//
> *:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/
> lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*:
> :/etc/tez/conf:/usr/lib/tez/*:/usr/lib/tez/lib/*:/usr/lib/
> hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/
> aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/
> share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ddb/lib/emr-
> ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-
> goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-
> hadoop.jar:/usr/share/aws/emr/cloudwatch-sink/lib/*:/usr/
> share/aws/emr/security/conf:/usr/share/aws/emr/security/
> lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-yarn/lib/*
>
>
>
> On Mon, Aug 21, 2017 at 11:21 AM, Steve Terrell <sterr...@oculus360.us>
> wrote:
>
>> Hmm... just repeated my test on emr-5.2.0. This time I went with the
>> default EMR console selections for master and core nodes (2 of them).
>>
>> When running my simple Pig Phoenix store script, still getting the errors
>> I got for other 5.x.x versions:
>> 2017-08-21 17:50:52,431 [ERROR] [main] |app.DAGAppMaster|: Error starting
>> DAGAppMaster
>> java.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.rec
>> ords.ContainerId.fromString(Ljava/lang/String;)Lorg/
>> apache/hadoop/yarn/api/records/ContainerId;
>> at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(Con
>> verterUtils.java:179)
>> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2304)
>>
>> The simple test script:
>> REGISTER /usr/lib/phoenix/phoenix-client.jar;
>> A = load '/steve/a.txt' as (TXT:chararray);
>> store A into 'hbase://A_TABLE' using org.apache.phoenix.pig.Phoenix
>> HBaseStorage('10.0.100.51','-batchSize 2500');
>>
>> Calling directly from the command line like
>> pig try.pig
>>
>> Maybe other people are calling their Phoenix Pig script some other way
>> (EMR steps) or with different parameters? Details where this works would
>> really help out a lot.
>>
>> Thanks,
>> Steve
>>
>> On Mon, Aug 21, 2017 at 10:23 AM, Steve Terrell <sterr...@oculus360.us>
>> wrote:
>>
>>> Anil,
>>>
>>> That's good news (about 5.2). Any chance I could see your
>>>
>>> - full pig command line
>>> - PIG_CLASSPATH env variable
>>> - pig script or at least the REGISTER and PhoenixHBaseStorage()
>>> lines?
>>>
>>> Might help me figure out what I'm doing wrong or differently.
>>>
>>> One thing I did not mention because I thought it should not matter is
>>> that to avoid extra costs while testing, I was only running a master node
>>> with no slaves (no task or core nodes). Maybe lack of slaves causes
>>> problems not normally seen. Interesting...
>>>
>>> Thanks so much,
>>> Steve
>>>
>>>
>>> On Sun, Aug 20, 2017 at 11:04 AM, anil gupta <anilgupt...@gmail.com>
>>> wrote:
>>>
>>>> Hey Steve,
>>>>
>>>> We are currently using EMR5.2 and pig-phoenix is working fine for us.
>>>> We are gonna try EMR5.8 next week.
>>>>
>>>> HTH,
>>>> Anil
>>>>
>>>> On Fri, Aug 18, 2017 at 9:00 AM, Steve Terrell <sterr...@oculus360.us>
>>>> wrote:
>>>>
>>>>> More info...
>>>>>
>>>>> By trial and error, I tested different EMR versions and made a little
>>>>> incomplete list of which ones support Pig Phoenix storage and which ones
>>>>> don't:
>>>>>
>>>>> emr-5.8.0 JacksonJaxbJsonProvider error
>>>>> emr-5.6.0 JacksonJaxbJsonProvider error
>>>>> emr-5.4.0 JacksonJaxbJsonProvider error
>>>>> emr-5.3.1 ContainerId.fromString() error
>>>>> emr-5.3.0 ContainerId.fromString() error
>>>>> emr-5.0.0 ContainerId.fromString() error
>>>>> emr-4.9.2 Works!
>>>>> emr-4.7.0 Works!
>>>>>
>>>>> I ran out of time trying to get 5.8.0 working, so will start using
>>>>> 4.9.2. But I would like to switch to 5.8.0 if anyone has a solution.
>>>>> Meanwhile, I hope this list saves other people some time and headache.
>>>>>
>>>>> Thanks,
>>>>> Steve
>>>>>
>>>>> On Thu, Aug 17, 2017 at 2:40 PM, Steve Terrell <sterr...@oculus360.us>
>>>>> wrote:
>>>>>
>>>>>> I'm running EMR 5.8.0 with these applications installed:
>>>>>> Pig 0.16.0, Phoenix 4.11.0, HBase 1.3.1
>>>>>>
>>>>>> Here is my pig script (try.pig):
>>>>>>
>>>>>> REGISTER /usr/lib/phoenix/phoenix-4.11.0-HBase-1.3-client.jar;
>>>>>> A = load '/steve/a.txt' as (TXT:chararray);
>>>>>> store A into 'hbase://A_TABLE' using org.apache.phoenix.pig.Phoenix
>>>>>> HBaseStorage('10.0.100.51','-batchSize 2500');
>>>>>>
>>>>>> I run it like this from the command line:
>>>>>> pig try.pig
>>>>>>
>>>>>> When it fails, I dig into the hadoop task logs and find this:
>>>>>> 2017-08-17 19:11:37,539 [ERROR] [main] |app.DAGAppMaster|: Error
>>>>>> starting DAGAppMaster
>>>>>> java.lang.NoClassDefFoundError: org/apache/phoenix/shaded/org/
>>>>>> codehaus/jackson/jaxrs/JacksonJaxbJsonProvider
>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>>>>> at java.security.SecureClassLoader.defineClass(SecureClassLoade
>>>>>> r.java:142)
>>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>> at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.se
>>>>>> rviceInit(TimelineClientImpl.java:269)
>>>>>> at org.apache.hadoop.service.AbstractService.init(AbstractServi
>>>>>> ce.java:163)
>>>>>> at org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingServ
>>>>>> ice.serviceInit(ATSHistoryLoggingService.java:102)
>>>>>> at org.apache.hadoop.service.AbstractService.init(AbstractServi
>>>>>> ce.java:163)
>>>>>> at org.apache.hadoop.service.CompositeService.serviceInit(Compo
>>>>>> siteService.java:107)
>>>>>> at org.apache.tez.dag.history.HistoryEventHandler.serviceInit(H
>>>>>> istoryEventHandler.java:73)
>>>>>> at org.apache.hadoop.service.AbstractService.init(AbstractServi
>>>>>> ce.java:163)
>>>>>> at org.apache.tez.dag.app.DAGAppMaster.initServices(DAGAppMaste
>>>>>> r.java:1922)
>>>>>> at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster
>>>>>> .java:624)
>>>>>> at org.apache.hadoop.service.AbstractService.init(AbstractServi
>>>>>> ce.java:163)
>>>>>> at org.apache.tez.dag.app.DAGAppMaster$8.run(DAGAppMaster.java:2557)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>>>>>> upInformation.java:1698)
>>>>>> at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DA
>>>>>> GAppMaster.java:2554)
>>>>>> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2359)
>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>> org.apache.phoenix.shaded.org.codehaus.jackson.jaxrs.Jackson
>>>>>> JaxbJsonProvider
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>> ... 28 more
>>>>>>
>>>>>> Has anyone been able to get org.apache.phoenix.pig.PhoenixHBaseStorage()
>>>>>> to work on recent EMR versions? Please help if you can.
>>>>>>
>>>>>> Thank you,
>>>>>> Steve
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Anil Gupta
>>>>
>>>
>>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
