Maybe this is something I can fix. If I were to add the cloudera/hortonworks maven repos, and then add some supported options for hadoop beyond 1/2, that would pretty much do it, right? ᐧ
On Tue, Aug 19, 2014 at 3:49 PM, Jesse Yates <[email protected]> wrote: > FWIW internally at Salesforce we also patch the HBase and Hadoop poms to > support our own internal 'light forks'. Its really not a big deal to manage > - a couple of jenkins jobs (one to automate, one to track open source > changes and ensure your patch(es) still work, etc) and you are good to go. > > I imagine this is also what various distributors are doing for their forks > as well. > > ------------------- > Jesse Yates > @jesse_yates > jyates.github.com > > > On Tue, Aug 19, 2014 at 3:36 PM, Russell Jurney <[email protected]> > wrote: > >> First of all, I apologize if you feel like I was picking on you. I was >> not trying to do that. >> >> My understanding is that Salesforce pays people to work on Phoenix. Is >> that not the case? I'm hoping one of them will add spark-like support for >> CDH and HDP to advance the project. >> >> And I don't mention the POM thing to pick on someone. The majority of >> HBase users are not going to be able to use Phoenix because they run a >> commercial distribution of Hadoop and aren't pom wizards. That seems kind >> of important for the well being of the project. >> ᐧ >> >> >> On Tue, Aug 19, 2014 at 3:26 PM, Andrew Purtell <[email protected]> >> wrote: >> >>> I don't think an Apache project should spend precious bandwidth tracking >>> the various and sundry redistributors of Apache ecosystem projects. This is >>> putting the cart before the horse. The horse is the Apache upstream >>> projects. The cart is the commercial distributions leveraging the Apache >>> ecosystem for profit. Spark is not a good example, it is supported by a >>> commercial concern, Databricks. What commercial company supports Phoenix? >>> Why do you think it is appropriate to pick on volunteers because editing >>> POM files is too much work? >>> >>> >>> On Tue, Aug 19, 2014 at 3:09 PM, Russell Jurney < >>> [email protected]> wrote: >>> >>>> I also created https://issues.apache.org/jira/browse/PHOENIX-1185 >>>> because requiring users to hand-edit the pom.xml just to build against CDH >>>> and HDP is nuts. >>>> ᐧ >>>> >>>> >>>> On Tue, Aug 19, 2014 at 3:03 PM, Russell Jurney < >>>> [email protected]> wrote: >>>> >>>>> I have created https://issues.apache.org/jira/browse/PHOENIX-1183 and >>>>> https://issues.apache.org/jira/browse/PHOENIX-1184 >>>>> >>>>> I hope this can be resolved soon, we are pretty screwed right now. >>>>> ᐧ >>>>> >>>>> >>>>> On Tue, Aug 19, 2014 at 2:08 PM, Jesse Yates <[email protected]> >>>>> wrote: >>>>> >>>>>> Yup, that looks like an issue to me :-/ >>>>>> >>>>>> ------------------- >>>>>> Jesse Yates >>>>>> @jesse_yates >>>>>> jyates.github.com >>>>>> >>>>>> >>>>>> On Tue, Aug 19, 2014 at 2:06 PM, Russell Jurney < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Running against any version would be ok, but it does not work. I get >>>>>>> this error: >>>>>>> >>>>>>> 2014-08-19 14:03:46,904 FATAL org.apache.hadoop.mapred.Child: Error >>>>>>> running child : java.lang.IncompatibleClassChangeError: Found interface >>>>>>> org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected >>>>>>> >>>>>>> ᐧ >>>>>>> I went back and checked out the Phoenix commit before the pom.xml >>>>>>> was upgraded to 0.98.4 >>>>>>> <https://github.com/apache/phoenix/commit/36a41c86a4cdb16ab91b95882dbdb02e6006831d>, >>>>>>> and I get the same error. >>>>>>> >>>>>>> Not sure what to do. Stuck. I imagine most users must be too, as CDH >>>>>>> 5 is probably the most popular installation there is? >>>>>>> >>>>>>> On Tue, Aug 19, 2014 at 11:50 AM, James Taylor < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> The dependencies on HBase 0.98.4 are *compile time* dependencies. >>>>>>>> Is it necessary for you to compile against CDH 5.1 or just run against >>>>>>>> it? >>>>>>>> >>>>>>>> >>>>>>>> On Tuesday, August 19, 2014, Russell Jurney < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Thats really bad. That means... CDH 5.x can't run Phoenix? How can >>>>>>>>> this be fixed? I'm not sure what to do. We're in limbo on our new >>>>>>>>> cluster >>>>>>>>> now. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Aug 18, 2014 at 11:57 PM, Ravi Kiran < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Russel, >>>>>>>>>> Apparently, Phoenix 4.0.0 leverages few API methods of HBase >>>>>>>>>> 0.98.4 v which aren't present within 0.98.1 that comes with CDH 5.1 >>>>>>>>>> . >>>>>>>>>> That's the primary cause for the build issues. >>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>> Ravi >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Aug 18, 2014 at 5:56 PM, Russell Jurney < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Talking to myself, but hopefully creating good docs. Replacing >>>>>>>>>>> the previous hadoop version with one I found here: >>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/, >>>>>>>>>>> 2.3.0-mr1-cdh5.1.0, >>>>>>>>>>> makes things get a little further. >>>>>>>>>>> >>>>>>>>>>> I can't get past some build errors, however. Has anyone done >>>>>>>>>>> this before me who can help? >>>>>>>>>>> >>>>>>>>>>> [ERROR] >>>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27] >>>>>>>>>>> cannot find symbol >>>>>>>>>>> >>>>>>>>>>> symbol: method valueOf(java.lang.String,int,int) >>>>>>>>>>> >>>>>>>>>>> location: class org.apache.hadoop.hbase.ServerName >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [ERROR] Failed to execute goal >>>>>>>>>>> org.apache.maven.plugins:maven-compiler-plugin:3.0:compile >>>>>>>>>>> (default-compile) on project phoenix-core: Compilation failure >>>>>>>>>>> >>>>>>>>>>> [ERROR] >>>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27] >>>>>>>>>>> cannot find symbol >>>>>>>>>>> >>>>>>>>>>> [ERROR] symbol: method valueOf(java.lang.String,int,int) >>>>>>>>>>> >>>>>>>>>>> [ERROR] location: class org.apache.hadoop.hbase.ServerName >>>>>>>>>>> >>>>>>>>>>> [ERROR] -> [Help 1] >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Aug 18, 2014 at 5:41 PM, Russell Jurney < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Ok, so it is clear to me what I have to do. I have to edit my >>>>>>>>>>>> pom.xml to point at CDH 5.1, which translates into: >>>>>>>>>>>> >>>>>>>>>>>> Add the cloudera repo: >>>>>>>>>>>> >>>>>>>>>>>> <repository> >>>>>>>>>>>> <id>cloudera</id> >>>>>>>>>>>> <url> >>>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/ >>>>>>>>>>>> </url> >>>>>>>>>>>> </repository> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Then change the hadoop and hbase versions: >>>>>>>>>>>> >>>>>>>>>>>> <!-- Hadoop Versions --> >>>>>>>>>>>> >>>>>>>>>>>> <hbase-hadoop1.version>0.98.1-cdh5.1.0</hbase-hadoop1.version> >>>>>>>>>>>> >>>>>>>>>>>> <hbase-hadoop2.version>0.98.1-cdh5.1.0</hbase-hadoop2.version> >>>>>>>>>>>> <hadoop-one.version>2.3.0-cdh5.1.0</hadoop-one.version> >>>>>>>>>>>> <hadoop-two.version>2.3.0-cdh5.1.0</hadoop-two.version> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> However, I get this error when I build, which tells me there is >>>>>>>>>>>> more complex POM surgery required. >>>>>>>>>>>> >>>>>>>>>>>> [ERROR] Failed to execute goal on project phoenix-core: Could >>>>>>>>>>>> not resolve dependencies for project >>>>>>>>>>>> org.apache.phoenix:phoenix-core:jar:4.0.0-incubating: The following >>>>>>>>>>>> artifacts could not be resolved: >>>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0, >>>>>>>>>>>> org.apache.hadoop:hadoop-test:jar:2.3.0-cdh5.1.0: Could not find >>>>>>>>>>>> artifact >>>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0 in apache release >>>>>>>>>>>> ( >>>>>>>>>>>> https://repository.apache.org/content/repositories/releases/) >>>>>>>>>>>> -> [Help 1] >>>>>>>>>>>> >>>>>>>>>>>> Beyond changing the versions, I do not know how to fix this. >>>>>>>>>>>> Can anyone help? >>>>>>>>>>>> >>>>>>>>>>>> In general, is it possible to actually handle different CDH >>>>>>>>>>>> versions in this project? One shouldn't have to do pom surgery to >>>>>>>>>>>> build >>>>>>>>>>>> Phoenix for the most common platform. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:15 PM, Russell Jurney < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> When I try to store data into Phoenix from Pig, I get this >>>>>>>>>>>>> error. I am on CDH 5.1, and Phoenix 4.0. >>>>>>>>>>>>> >>>>>>>>>>>>> Anyone know how to resolve this issue? >>>>>>>>>>>>> >>>>>>>>>>>>> 2014-08-18 17:11:25,165 INFO >>>>>>>>>>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader: >>>>>>>>>>>>> Current split being processed >>>>>>>>>>>>> hdfs://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+845 >>>>>>>>>>>>> 2014-08-18 >>>>>>>>>>>>> <http://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+8452014-08-18> >>>>>>>>>>>>> 17:11:25,173 INFO org.apache.hadoop.mapred.TaskLogsTruncater: >>>>>>>>>>>>> Initializing logs' truncater with mapRetainSize=-1 and >>>>>>>>>>>>> reduceRetainSize=-1 >>>>>>>>>>>>> 2014-08-18 17:11:25,175 FATAL org.apache.hadoop.mapred.Child: >>>>>>>>>>>>> Error running child : java.lang.IncompatibleClassChangeError: >>>>>>>>>>>>> Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, >>>>>>>>>>>>> but class was expected >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.phoenix.pig.hadoop.PhoenixOutputFormat.getRecordWriter(PhoenixOutputFormat.java:65) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:548) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:653) >>>>>>>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) >>>>>>>>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) >>>>>>>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:262) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Russell Jurney twitter.com/rjurney [email protected] >>>>>>>>>>>>> datasyndrome.com >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Russell Jurney twitter.com/rjurney [email protected] >>>>>>>>>>>> datasyndrome.com >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Russell Jurney twitter.com/rjurney [email protected] >>>>>>>>>>> datasyndrome.com >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Russell Jurney twitter.com/rjurney [email protected] >>>>>>>>> datasyndrome.com >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Russell Jurney twitter.com/rjurney [email protected] >>>>>>> datasyndrome.com >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Russell Jurney twitter.com/rjurney [email protected] >>>>> datasyndrome.com >>>>> >>>> >>>> >>>> >>>> -- >>>> Russell Jurney twitter.com/rjurney [email protected] >>>> datasyndrome.com >>>> >>> >>> >>> >>> -- >>> Best regards, >>> >>> - Andy >>> >>> Problems worthy of attack prove their worth by hitting back. - Piet Hein >>> (via Tom White) >>> >> >> >> >> -- >> Russell Jurney twitter.com/rjurney [email protected] datasyndrome. >> com >> > > -- Russell Jurney twitter.com/rjurney [email protected] datasyndrome.com
