date:20170821

Re: quick question about apache phoenix.

2017-08-21 Thread Jonathan Leech

I recognize that name. Some of his posts were... memorable. I'm not surprised 
to hear he was banned. 

> On Aug 21, 2017, at 11:06 PM, James Taylor  wrote:
> 
> Hi Pawan,
> Why would you listen to someone about the future of Apache Phoenix who has no 
> involvement in or knowledge about the project? Michael Segel was banned on 
> the HBase lists for a while due to being too belligerent. He's not a 
> committer on any Apache project. Has he even contributed a patch? To any 
> project? Ever?
> 
> You're welcome to join our dev and user list to get an idea of what we're up 
> to.
> 
> Thanks,
> James
> 
>> On Mon, Aug 21, 2017 at 9:40 PM, Agnihotri, Pawan  
>> wrote:
>> Hello Phoenix group,
>> 
>>  
>> 
>> There was some doubt on phoenix project closing down in future so wanted to 
>> confirm with you.  I don’t see that coming but It will be helpful to have 
>> your views. Thank you.
>> 
>>  
>> 
>> https://community.mapr.com/thread/22104-apache-phoenix-support-for-mapr-5051-and-52
>> 
>>  
>> 
>> 
>> 
>>  
>> 
>>  
>> 
>> Thanks & Regards,
>> 
>> Pawan Agnihotri
>> 
>> Software Developer, Business Technology Solutions
>> Valassis Direct Mail, Inc
>> 1 Targeting Center
>> 
>> Windsor CT 06095
>> 
>> Phone: +1 860 298 5520
>> 
>> Cell: +1 860 325 8794
>> 
>> India : +91 775 708 1071
>> agnihot...@valassis.com
>> www.valassis.com
>> 
>>  
>> 
>

Re: quick question about apache phoenix.

2017-08-21 Thread James Taylor

Hi Pawan,
Why would you listen to someone about the future of Apache Phoenix who has
no involvement in or knowledge about the project? Michael Segel was banned
on the HBase lists for a while due to being too belligerent. He's not a
committer on any Apache project. Has he even contributed a patch? To any
project? Ever?

You're welcome to join our dev and user list to get an idea of what we're
up to.

Thanks,
James

On Mon, Aug 21, 2017 at 9:40 PM, Agnihotri, Pawan 
wrote:

> Hello Phoenix group,
>
>
>
> There was some doubt on phoenix project closing down in future so wanted
> to confirm with you.  I don’t see that coming but It will be helpful to
> have your views. Thank you.
>
>
>
> https://community.mapr.com/thread/22104-apache-phoenix-
> support-for-mapr-5051-and-52
>
>
>
>
>
>
>
> Thanks & Regards,
>
> Pawan Agnihotri
>
> Software Developer, Business Technology Solutions
> Valassis Direct Mail, Inc
> 1 Targeting Center
>
> Windsor CT 06095
>
> Phone: +1 860 298 5520 <(860)%20298-5520>
>
> Cell: +1 860 325 8794 <(860)%20325-8794>
>
> India : +91 775 708 1071 <+91%2077570%2081071>
> agnihot...@valassis.com
> www.valassis.com
>
>
>

quick question about apache phoenix.

2017-08-21 Thread Agnihotri, Pawan

Hello Phoenix group,

There was some doubt on phoenix project closing down in future so wanted to 
confirm with you.  I don't see that coming but It will be helpful to have your 
views. Thank you.

https://community.mapr.com/thread/22104-apache-phoenix-support-for-mapr-5051-and-52

[cid:image001.png@01D31ADF.50A2C4B0]


Thanks & Regards,
Pawan Agnihotri
Software Developer, Business Technology Solutions
Valassis Direct Mail, Inc
1 Targeting Center
Windsor CT 06095
Phone: +1 860 298 5520
Cell: +1 860 325 8794
India : +91 775 708 1071
agnihot...@valassis.com
www.valassis.com

Re: Phoenix Storage Not Working on AWS EMR 5.8.0

2017-08-21 Thread Steve Terrell

Thanks for the extra info!  Will let everyone know if I solve this.

On Mon, Aug 21, 2017 at 4:24 PM, anil gupta  wrote:

> And forgot to mention that we invoke our pig scripts through oozie.
>
> On Mon, Aug 21, 2017 at 2:20 PM, anil gupta  wrote:
>
>> Sorry, cant share the pig script.
>> Here is what we are registering:
>> REGISTER /usr/lib/phoenix/phoenix-4.7.0-HBase-1.2-client.jar;
>> REGISTER /usr/lib/pig/lib/piggybank.jar;
>>
>>
>> Following is the classpath of Hadoop and Yarn:
>> [hadoop@ip-52-143 ~]$ hadoop classpath
>> /etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:
>> /usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/
>> lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/
>> hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/
>> hadoop-mapreduce/.//*::/etc/tez/conf:/usr/lib/tez/*:/usr/
>> lib/tez/lib/*:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-
>> java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/
>> emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/
>> aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/
>> goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/
>> kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/aws/emr/cloudw
>> atch-sink/lib/*:/usr/share/aws/emr/security/conf:/usr/
>> share/aws/emr/security/lib/*
>> [hadoop@ip-52-143 ~]$ yarn classpath
>> /etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/
>> hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/
>> usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/
>> lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/
>> hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*::/
>> etc/tez/conf:/usr/lib/tez/*:/usr/lib/tez/lib/*:/usr/lib/had
>> oop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/
>> emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/
>> aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-
>> hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies
>> .jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop
>> .jar:/usr/share/aws/emr/cloudwatch-sink/lib/*:/usr/share/
>> aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/
>> usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-yarn/lib/*
>>
>>
>>
>> On Mon, Aug 21, 2017 at 11:21 AM, Steve Terrell 
>> wrote:
>>
>>> Hmm...  just repeated my test on emr-5.2.0.  This time I went with the
>>> default EMR console selections for master and core nodes (2 of them).
>>>
>>> When running my simple Pig Phoenix store script, still getting the
>>> errors I got for other 5.x.x versions:
>>> 2017-08-21 17:50:52,431 [ERROR] [main] |app.DAGAppMaster|: Error
>>> starting DAGAppMaster
>>> java.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.rec
>>> ords.ContainerId.fromString(Ljava/lang/String;)Lorg/apache/
>>> hadoop/yarn/api/records/ContainerId;
>>> at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(Con
>>> verterUtils.java:179)
>>> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2304)
>>>
>>> The simple test script:
>>> REGISTER /usr/lib/phoenix/phoenix-client.jar;
>>> A = load '/steve/a.txt' as (TXT:chararray);
>>> store A into 'hbase://A_TABLE' using org.apache.phoenix.pig.Phoenix
>>> HBaseStorage('10.0.100.51','-batchSize 2500');
>>>
>>> Calling directly from the command line like
>>> pig try.pig
>>>
>>> Maybe other people are calling their Phoenix Pig script some other way
>>> (EMR steps) or with different parameters?  Details where this works would
>>> really help out a lot.
>>>
>>> Thanks,
>>> Steve
>>>
>>> On Mon, Aug 21, 2017 at 10:23 AM, Steve Terrell 
>>> wrote:
>>>
 Anil,

 That's good news (about 5.2).  Any chance I could see your

- full pig command line
- PIG_CLASSPATH env variable
- pig script or at least the REGISTER and PhoenixHBaseStorage()
lines?

 Might help me figure out what I'm doing wrong or differently.

 One thing I did not mention because I thought it should not matter is
 that to avoid extra costs while testing, I was only running a master node
 with no slaves (no task or core nodes).  Maybe lack of slaves causes
 problems not normally seen.  Interesting...

 Thanks so much,
 Steve


 On Sun, Aug 20, 2017 at 11:04 AM, anil gupta 
 wrote:

> Hey Steve,
>
> We are currently using EMR5.2 and pig-phoenix is working fine for us.
> We are gonna try EMR5.8 next week.
>
> HTH,
> Anil
>
> On Fri, Aug 18, 2017 at 9:00 AM, Steve Terrell 
> wrote:
>
>> More info...
>>
>> By trial and error, I tested different EMR versions and made a little
>> incomplete list of which ones support Pig Phoenix storage and which ones
>> don't:
>>
>> emr-5.8.0 JacksonJaxbJsonProvider error
>> emr-5.6.0 JacksonJaxbJsonProvider error
>> emr-5.4.0

Re: Phoenix Storage Not Working on AWS EMR 5.8.0

2017-08-21 Thread anil gupta

And forgot to mention that we invoke our pig scripts through oozie.

On Mon, Aug 21, 2017 at 2:20 PM, anil gupta  wrote:

> Sorry, cant share the pig script.
> Here is what we are registering:
> REGISTER /usr/lib/phoenix/phoenix-4.7.0-HBase-1.2-client.jar;
> REGISTER /usr/lib/pig/lib/piggybank.jar;
>
>
> Following is the classpath of Hadoop and Yarn:
> [hadoop@ip-52-143 ~]$ hadoop classpath
> /etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.
> //*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/
> usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/
> lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/
> lib/hadoop-mapreduce/.//*::/etc/tez/conf:/usr/lib/tez/*:/
> usr/lib/tez/lib/*:/usr/lib/hadoop-lzo/lib/*:/usr/share/
> aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/
> aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/
> usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/
> emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/
> emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/aws/emr/
> cloudwatch-sink/lib/*:/usr/share/aws/emr/security/conf:/
> usr/share/aws/emr/security/lib/*
> [hadoop@ip-52-143 ~]$ yarn classpath
> /etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/
> lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-
> hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//
> *:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/
> lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*:
> :/etc/tez/conf:/usr/lib/tez/*:/usr/lib/tez/lib/*:/usr/lib/
> hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/
> aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/
> share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ddb/lib/emr-
> ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-
> goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-
> hadoop.jar:/usr/share/aws/emr/cloudwatch-sink/lib/*:/usr/
> share/aws/emr/security/conf:/usr/share/aws/emr/security/
> lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-yarn/lib/*
>
>
>
> On Mon, Aug 21, 2017 at 11:21 AM, Steve Terrell 
> wrote:
>
>> Hmm...  just repeated my test on emr-5.2.0.  This time I went with the
>> default EMR console selections for master and core nodes (2 of them).
>>
>> When running my simple Pig Phoenix store script, still getting the errors
>> I got for other 5.x.x versions:
>> 2017-08-21 17:50:52,431 [ERROR] [main] |app.DAGAppMaster|: Error starting
>> DAGAppMaster
>> java.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.rec
>> ords.ContainerId.fromString(Ljava/lang/String;)Lorg/
>> apache/hadoop/yarn/api/records/ContainerId;
>> at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(Con
>> verterUtils.java:179)
>> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2304)
>>
>> The simple test script:
>> REGISTER /usr/lib/phoenix/phoenix-client.jar;
>> A = load '/steve/a.txt' as (TXT:chararray);
>> store A into 'hbase://A_TABLE' using org.apache.phoenix.pig.Phoenix
>> HBaseStorage('10.0.100.51','-batchSize 2500');
>>
>> Calling directly from the command line like
>> pig try.pig
>>
>> Maybe other people are calling their Phoenix Pig script some other way
>> (EMR steps) or with different parameters?  Details where this works would
>> really help out a lot.
>>
>> Thanks,
>> Steve
>>
>> On Mon, Aug 21, 2017 at 10:23 AM, Steve Terrell 
>> wrote:
>>
>>> Anil,
>>>
>>> That's good news (about 5.2).  Any chance I could see your
>>>
>>>- full pig command line
>>>- PIG_CLASSPATH env variable
>>>- pig script or at least the REGISTER and PhoenixHBaseStorage()
>>>lines?
>>>
>>> Might help me figure out what I'm doing wrong or differently.
>>>
>>> One thing I did not mention because I thought it should not matter is
>>> that to avoid extra costs while testing, I was only running a master node
>>> with no slaves (no task or core nodes).  Maybe lack of slaves causes
>>> problems not normally seen.  Interesting...
>>>
>>> Thanks so much,
>>> Steve
>>>
>>>
>>> On Sun, Aug 20, 2017 at 11:04 AM, anil gupta 
>>> wrote:
>>>
 Hey Steve,

 We are currently using EMR5.2 and pig-phoenix is working fine for us.
 We are gonna try EMR5.8 next week.

 HTH,
 Anil

 On Fri, Aug 18, 2017 at 9:00 AM, Steve Terrell 
 wrote:

> More info...
>
> By trial and error, I tested different EMR versions and made a little
> incomplete list of which ones support Pig Phoenix storage and which ones
> don't:
>
> emr-5.8.0 JacksonJaxbJsonProvider error
> emr-5.6.0 JacksonJaxbJsonProvider error
> emr-5.4.0 JacksonJaxbJsonProvider error
> emr-5.3.1 ContainerId.fromString() error
> emr-5.3.0 ContainerId.fromString() error
> emr-5.0.0 ContainerId.fromString() error
> emr-4.9.2 Works!
> emr-4.7.0 Works!
>
> I ran out of time trying to get 5.8.0 working, so

Re: Phoenix Storage Not Working on AWS EMR 5.8.0

2017-08-21 Thread anil gupta

Sorry, cant share the pig script.
Here is what we are registering:
REGISTER /usr/lib/phoenix/phoenix-4.7.0-HBase-1.2-client.jar;
REGISTER /usr/lib/pig/lib/piggybank.jar;


Following is the classpath of Hadoop and Yarn:
[hadoop@ip-52-143 ~]$ hadoop classpath
/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*::/etc/tez/conf:/usr/lib/tez/*:/usr/lib/tez/lib/*:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/aws/emr/cloudwatch-sink/lib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*
[hadoop@ip-52-143 ~]$ yarn classpath
/etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*::/etc/tez/conf:/usr/lib/tez/*:/usr/lib/tez/lib/*:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/aws/emr/cloudwatch-sink/lib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-yarn/lib/*



On Mon, Aug 21, 2017 at 11:21 AM, Steve Terrell 
wrote:

> Hmm...  just repeated my test on emr-5.2.0.  This time I went with the
> default EMR console selections for master and core nodes (2 of them).
>
> When running my simple Pig Phoenix store script, still getting the errors
> I got for other 5.x.x versions:
> 2017-08-21 17:50:52,431 [ERROR] [main] |app.DAGAppMaster|: Error starting
> DAGAppMaster
> java.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.
> records.ContainerId.fromString(Ljava/lang/String;)
> Lorg/apache/hadoop/yarn/api/records/ContainerId;
> at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(
> ConverterUtils.java:179)
> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2304)
>
> The simple test script:
> REGISTER /usr/lib/phoenix/phoenix-client.jar;
> A = load '/steve/a.txt' as (TXT:chararray);
> store A into 'hbase://A_TABLE' using org.apache.phoenix.pig.
> PhoenixHBaseStorage('10.0.100.51','-batchSize 2500');
>
> Calling directly from the command line like
> pig try.pig
>
> Maybe other people are calling their Phoenix Pig script some other way
> (EMR steps) or with different parameters?  Details where this works would
> really help out a lot.
>
> Thanks,
> Steve
>
> On Mon, Aug 21, 2017 at 10:23 AM, Steve Terrell 
> wrote:
>
>> Anil,
>>
>> That's good news (about 5.2).  Any chance I could see your
>>
>>- full pig command line
>>- PIG_CLASSPATH env variable
>>- pig script or at least the REGISTER and PhoenixHBaseStorage() lines?
>>
>> Might help me figure out what I'm doing wrong or differently.
>>
>> One thing I did not mention because I thought it should not matter is
>> that to avoid extra costs while testing, I was only running a master node
>> with no slaves (no task or core nodes).  Maybe lack of slaves causes
>> problems not normally seen.  Interesting...
>>
>> Thanks so much,
>> Steve
>>
>>
>> On Sun, Aug 20, 2017 at 11:04 AM, anil gupta 
>> wrote:
>>
>>> Hey Steve,
>>>
>>> We are currently using EMR5.2 and pig-phoenix is working fine for us. We
>>> are gonna try EMR5.8 next week.
>>>
>>> HTH,
>>> Anil
>>>
>>> On Fri, Aug 18, 2017 at 9:00 AM, Steve Terrell 
>>> wrote:
>>>
 More info...

 By trial and error, I tested different EMR versions and made a little
 incomplete list of which ones support Pig Phoenix storage and which ones
 don't:

 emr-5.8.0 JacksonJaxbJsonProvider error
 emr-5.6.0 JacksonJaxbJsonProvider error
 emr-5.4.0 JacksonJaxbJsonProvider error
 emr-5.3.1 ContainerId.fromString() error
 emr-5.3.0 ContainerId.fromString() error
 emr-5.0.0 ContainerId.fromString() error
 emr-4.9.2 Works!
 emr-4.7.0 Works!

 I ran out of time trying to get 5.8.0 working, so will start using
 4.9.2.  But I would like to switch to 5.8.0 if anyone has a solution.
 Meanwhile, I hope this list saves other people some time and headache.

 Thanks,
 Steve

 On Thu, Aug 17, 2017 at 2:40 PM, Steve Terrell 
 wrote:

> I'm running EMR 5.8.0 with these

Re: Phoenix Storage Not Working on AWS EMR 5.8.0

2017-08-21 Thread Steve Terrell

Hmm...  just repeated my test on emr-5.2.0.  This time I went with the
default EMR console selections for master and core nodes (2 of them).

When running my simple Pig Phoenix store script, still getting the errors I
got for other 5.x.x versions:
2017-08-21 17:50:52,431 [ERROR] [main] |app.DAGAppMaster|: Error starting
DAGAppMaster
java.lang.NoSuchMethodError:
org.apache.hadoop.yarn.api.records.ContainerId.fromString(Ljava/lang/String;)Lorg/apache/hadoop/yarn/api/records/ContainerId;
at
org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:179)
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2304)

The simple test script:
REGISTER /usr/lib/phoenix/phoenix-client.jar;
A = load '/steve/a.txt' as (TXT:chararray);
store A into 'hbase://A_TABLE' using
org.apache.phoenix.pig.PhoenixHBaseStorage('10.0.100.51','-batchSize 2500');

Calling directly from the command line like
pig try.pig

Maybe other people are calling their Phoenix Pig script some other way (EMR
steps) or with different parameters?  Details where this works would really
help out a lot.

Thanks,
Steve

On Mon, Aug 21, 2017 at 10:23 AM, Steve Terrell 
wrote:

> Anil,
>
> That's good news (about 5.2).  Any chance I could see your
>
>- full pig command line
>- PIG_CLASSPATH env variable
>- pig script or at least the REGISTER and PhoenixHBaseStorage() lines?
>
> Might help me figure out what I'm doing wrong or differently.
>
> One thing I did not mention because I thought it should not matter is that
> to avoid extra costs while testing, I was only running a master node with
> no slaves (no task or core nodes).  Maybe lack of slaves causes problems
> not normally seen.  Interesting...
>
> Thanks so much,
> Steve
>
>
> On Sun, Aug 20, 2017 at 11:04 AM, anil gupta 
> wrote:
>
>> Hey Steve,
>>
>> We are currently using EMR5.2 and pig-phoenix is working fine for us. We
>> are gonna try EMR5.8 next week.
>>
>> HTH,
>> Anil
>>
>> On Fri, Aug 18, 2017 at 9:00 AM, Steve Terrell 
>> wrote:
>>
>>> More info...
>>>
>>> By trial and error, I tested different EMR versions and made a little
>>> incomplete list of which ones support Pig Phoenix storage and which ones
>>> don't:
>>>
>>> emr-5.8.0 JacksonJaxbJsonProvider error
>>> emr-5.6.0 JacksonJaxbJsonProvider error
>>> emr-5.4.0 JacksonJaxbJsonProvider error
>>> emr-5.3.1 ContainerId.fromString() error
>>> emr-5.3.0 ContainerId.fromString() error
>>> emr-5.0.0 ContainerId.fromString() error
>>> emr-4.9.2 Works!
>>> emr-4.7.0 Works!
>>>
>>> I ran out of time trying to get 5.8.0 working, so will start using
>>> 4.9.2.  But I would like to switch to 5.8.0 if anyone has a solution.
>>> Meanwhile, I hope this list saves other people some time and headache.
>>>
>>> Thanks,
>>> Steve
>>>
>>> On Thu, Aug 17, 2017 at 2:40 PM, Steve Terrell 
>>> wrote:
>>>
 I'm running EMR 5.8.0 with these applications installed:
 Pig 0.16.0, Phoenix 4.11.0, HBase 1.3.1

 Here is my pig script (try.pig):

 REGISTER /usr/lib/phoenix/phoenix-4.11.0-HBase-1.3-client.jar;
 A = load '/steve/a.txt' as (TXT:chararray);
 store A into 'hbase://A_TABLE' using org.apache.phoenix.pig.Phoenix
 HBaseStorage('10.0.100.51','-batchSize 2500');

 I run it like this from the command line:
 pig try.pig

 When it fails, I dig into the hadoop task logs and find this:
 2017-08-17 19:11:37,539 [ERROR] [main] |app.DAGAppMaster|: Error
 starting DAGAppMaster
 java.lang.NoClassDefFoundError: org/apache/phoenix/shaded/org/
 codehaus/jackson/jaxrs/JacksonJaxbJsonProvider
 at java.lang.ClassLoader.defineClass1(Native Method)
 at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
 at java.security.SecureClassLoader.defineClass(SecureClassLoade
 r.java:142)
 at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
 at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
 at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
 at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.se
 rviceInit(TimelineClientImpl.java:269)
 at org.apache.hadoop.service.AbstractService.init(AbstractServi
 ce.java:163)
 at org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingServ
 ice.serviceInit(ATSHistoryLoggingService.java:102)
 at org.apache.hadoop.service.AbstractService.init(AbstractServi
 ce.java:163)
 at org.apache.hadoop.service.CompositeService.serviceInit(Compo

Re: Phoenix Storage Not Working on AWS EMR 5.8.0

2017-08-21 Thread Steve Terrell

Anil,

That's good news (about 5.2).  Any chance I could see your

   - full pig command line
   - PIG_CLASSPATH env variable
   - pig script or at least the REGISTER and PhoenixHBaseStorage() lines?

Might help me figure out what I'm doing wrong or differently.

One thing I did not mention because I thought it should not matter is that
to avoid extra costs while testing, I was only running a master node with
no slaves (no task or core nodes).  Maybe lack of slaves causes problems
not normally seen.  Interesting...

Thanks so much,
Steve


On Sun, Aug 20, 2017 at 11:04 AM, anil gupta  wrote:

> Hey Steve,
>
> We are currently using EMR5.2 and pig-phoenix is working fine for us. We
> are gonna try EMR5.8 next week.
>
> HTH,
> Anil
>
> On Fri, Aug 18, 2017 at 9:00 AM, Steve Terrell 
> wrote:
>
>> More info...
>>
>> By trial and error, I tested different EMR versions and made a little
>> incomplete list of which ones support Pig Phoenix storage and which ones
>> don't:
>>
>> emr-5.8.0 JacksonJaxbJsonProvider error
>> emr-5.6.0 JacksonJaxbJsonProvider error
>> emr-5.4.0 JacksonJaxbJsonProvider error
>> emr-5.3.1 ContainerId.fromString() error
>> emr-5.3.0 ContainerId.fromString() error
>> emr-5.0.0 ContainerId.fromString() error
>> emr-4.9.2 Works!
>> emr-4.7.0 Works!
>>
>> I ran out of time trying to get 5.8.0 working, so will start using
>> 4.9.2.  But I would like to switch to 5.8.0 if anyone has a solution.
>> Meanwhile, I hope this list saves other people some time and headache.
>>
>> Thanks,
>> Steve
>>
>> On Thu, Aug 17, 2017 at 2:40 PM, Steve Terrell 
>> wrote:
>>
>>> I'm running EMR 5.8.0 with these applications installed:
>>> Pig 0.16.0, Phoenix 4.11.0, HBase 1.3.1
>>>
>>> Here is my pig script (try.pig):
>>>
>>> REGISTER /usr/lib/phoenix/phoenix-4.11.0-HBase-1.3-client.jar;
>>> A = load '/steve/a.txt' as (TXT:chararray);
>>> store A into 'hbase://A_TABLE' using org.apache.phoenix.pig.Phoenix
>>> HBaseStorage('10.0.100.51','-batchSize 2500');
>>>
>>> I run it like this from the command line:
>>> pig try.pig
>>>
>>> When it fails, I dig into the hadoop task logs and find this:
>>> 2017-08-17 19:11:37,539 [ERROR] [main] |app.DAGAppMaster|: Error
>>> starting DAGAppMaster
>>> java.lang.NoClassDefFoundError: org/apache/phoenix/shaded/org/
>>> codehaus/jackson/jaxrs/JacksonJaxbJsonProvider
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>> at java.security.SecureClassLoader.defineClass(SecureClassLoade
>>> r.java:142)
>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>> at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.se
>>> rviceInit(TimelineClientImpl.java:269)
>>> at org.apache.hadoop.service.AbstractService.init(AbstractServi
>>> ce.java:163)
>>> at org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingServ
>>> ice.serviceInit(ATSHistoryLoggingService.java:102)
>>> at org.apache.hadoop.service.AbstractService.init(AbstractServi
>>> ce.java:163)
>>> at org.apache.hadoop.service.CompositeService.serviceInit(Compo
>>> siteService.java:107)
>>> at org.apache.tez.dag.history.HistoryEventHandler.serviceInit(H
>>> istoryEventHandler.java:73)
>>> at org.apache.hadoop.service.AbstractService.init(AbstractServi
>>> ce.java:163)
>>> at org.apache.tez.dag.app.DAGAppMaster.initServices(DAGAppMaste
>>> r.java:1922)
>>> at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster
>>> .java:624)
>>> at org.apache.hadoop.service.AbstractService.init(AbstractServi
>>> ce.java:163)
>>> at org.apache.tez.dag.app.DAGAppMaster$8.run(DAGAppMaster.java:2557)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>>> upInformation.java:1698)
>>> at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DA
>>> GAppMaster.java:2554)
>>> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2359)
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.phoenix.shaded.org.codehaus.jackson.jaxrs.Jackson
>>> JaxbJsonProvider
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>> ... 28 more

Re: quick question about apache phoenix.

Re: quick question about apache phoenix.

quick question about apache phoenix.

Re: Phoenix Storage Not Working on AWS EMR 5.8.0

Re: Phoenix Storage Not Working on AWS EMR 5.8.0

Re: Phoenix Storage Not Working on AWS EMR 5.8.0

Re: Phoenix Storage Not Working on AWS EMR 5.8.0

Re: Phoenix Storage Not Working on AWS EMR 5.8.0

8 matches

Site Navigation

Mail list logo

Footer information