I agree the vendor should resolve these issues. Hortonworks has already included Phoenix in HDP. Cloudera is behind the curve here. I'm told they'll include Phoenix when they feel they can support it well.
That being said, wouldn't adding CDH/HDP options in pom.xml make the project easier to use, and result in more people trying to use CDH/HDP with Phoenix (and more peiople using Phoenix in general), which would bring up bugs like the ones here? Ideally the vendors would fix these JIRAs. That would seem to be a good thing. ᐧ On Tue, Aug 19, 2014 at 5:34 PM, Andrew Purtell <[email protected]> wrote: > Maybe "pick on" didn't get close enough to what I was after. > > > Maybe this is something I can fix. If I were to add the > cloudera/hortonworks maven repos, and then add some supported options for > hadoop beyond 1/2, that would pretty much do it, right? > > I doubt it, because v4 and master branches probably won't compile against > either, certainly not against CDH 5.1, their HBase releases are out of step > and stale with respect to the latest Apache HBase 0.98 and Apache Phoenix 4 > releases. > > Getting back to my point, it's unfair in my opinion to expect the upstream > volunteer Apache projects to track all of the commercial options and the > vagaries of their arbitrary code freezes and "curated" additional patches. > It's unfair to expect Salesforce to fund such an effort, unless Salesforce > has somehow gone into the Hadoop distribution business. Certainly I am not > speaking on behalf of Salesforce or anyone else here. On the other hand, I > think it would be totally reasonable to request your favorite vendor > address Phoenix related issues with *their* derivative distributions. > > > > On Tue, Aug 19, 2014 at 3:53 PM, Russell Jurney <[email protected]> > wrote: > >> Maybe this is something I can fix. If I were to add the >> cloudera/hortonworks maven repos, and then add some supported options for >> hadoop beyond 1/2, that would pretty much do it, right? >> ᐧ >> >> >> On Tue, Aug 19, 2014 at 3:49 PM, Jesse Yates <[email protected]> >> wrote: >> >>> FWIW internally at Salesforce we also patch the HBase and Hadoop poms to >>> support our own internal 'light forks'. Its really not a big deal to manage >>> - a couple of jenkins jobs (one to automate, one to track open source >>> changes and ensure your patch(es) still work, etc) and you are good to go. >>> >>> I imagine this is also what various distributors are doing for their >>> forks as well. >>> >>> ------------------- >>> Jesse Yates >>> @jesse_yates >>> jyates.github.com >>> >>> >>> On Tue, Aug 19, 2014 at 3:36 PM, Russell Jurney < >>> [email protected]> wrote: >>> >>>> First of all, I apologize if you feel like I was picking on you. I was >>>> not trying to do that. >>>> >>>> My understanding is that Salesforce pays people to work on Phoenix. Is >>>> that not the case? I'm hoping one of them will add spark-like support for >>>> CDH and HDP to advance the project. >>>> >>>> And I don't mention the POM thing to pick on someone. The majority of >>>> HBase users are not going to be able to use Phoenix because they run a >>>> commercial distribution of Hadoop and aren't pom wizards. That seems kind >>>> of important for the well being of the project. >>>> ᐧ >>>> >>>> >>>> On Tue, Aug 19, 2014 at 3:26 PM, Andrew Purtell <[email protected]> >>>> wrote: >>>> >>>>> I don't think an Apache project should spend precious bandwidth >>>>> tracking the various and sundry redistributors of Apache ecosystem >>>>> projects. This is putting the cart before the horse. The horse is the >>>>> Apache upstream projects. The cart is the commercial distributions >>>>> leveraging the Apache ecosystem for profit. Spark is not a good example, >>>>> it >>>>> is supported by a commercial concern, Databricks. What commercial company >>>>> supports Phoenix? Why do you think it is appropriate to pick on volunteers >>>>> because editing POM files is too much work? >>>>> >>>>> >>>>> On Tue, Aug 19, 2014 at 3:09 PM, Russell Jurney < >>>>> [email protected]> wrote: >>>>> >>>>>> I also created https://issues.apache.org/jira/browse/PHOENIX-1185 >>>>>> because requiring users to hand-edit the pom.xml just to build against >>>>>> CDH >>>>>> and HDP is nuts. >>>>>> ᐧ >>>>>> >>>>>> >>>>>> On Tue, Aug 19, 2014 at 3:03 PM, Russell Jurney < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> I have created https://issues.apache.org/jira/browse/PHOENIX-1183 >>>>>>> and https://issues.apache.org/jira/browse/PHOENIX-1184 >>>>>>> >>>>>>> I hope this can be resolved soon, we are pretty screwed right now. >>>>>>> ᐧ >>>>>>> >>>>>>> >>>>>>> On Tue, Aug 19, 2014 at 2:08 PM, Jesse Yates < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Yup, that looks like an issue to me :-/ >>>>>>>> >>>>>>>> ------------------- >>>>>>>> Jesse Yates >>>>>>>> @jesse_yates >>>>>>>> jyates.github.com >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Aug 19, 2014 at 2:06 PM, Russell Jurney < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Running against any version would be ok, but it does not work. I >>>>>>>>> get this error: >>>>>>>>> >>>>>>>>> 2014-08-19 14:03:46,904 FATAL org.apache.hadoop.mapred.Child: Error >>>>>>>>> running child : java.lang.IncompatibleClassChangeError: Found >>>>>>>>> interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class >>>>>>>>> was expected >>>>>>>>> >>>>>>>>> ᐧ >>>>>>>>> I went back and checked out the Phoenix commit before the pom.xml >>>>>>>>> was upgraded to 0.98.4 >>>>>>>>> <https://github.com/apache/phoenix/commit/36a41c86a4cdb16ab91b95882dbdb02e6006831d>, >>>>>>>>> and I get the same error. >>>>>>>>> >>>>>>>>> Not sure what to do. Stuck. I imagine most users must be too, as >>>>>>>>> CDH 5 is probably the most popular installation there is? >>>>>>>>> >>>>>>>>> On Tue, Aug 19, 2014 at 11:50 AM, James Taylor < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> The dependencies on HBase 0.98.4 are *compile time* >>>>>>>>>> dependencies. Is it necessary for you to compile against CDH 5.1 or >>>>>>>>>> just >>>>>>>>>> run against it? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tuesday, August 19, 2014, Russell Jurney < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Thats really bad. That means... CDH 5.x can't run Phoenix? How >>>>>>>>>>> can this be fixed? I'm not sure what to do. We're in limbo on our >>>>>>>>>>> new >>>>>>>>>>> cluster now. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Aug 18, 2014 at 11:57 PM, Ravi Kiran < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Russel, >>>>>>>>>>>> Apparently, Phoenix 4.0.0 leverages few API methods of >>>>>>>>>>>> HBase 0.98.4 v which aren't present within 0.98.1 that comes with >>>>>>>>>>>> CDH 5.1 >>>>>>>>>>>> . That's the primary cause for the build issues. >>>>>>>>>>>> >>>>>>>>>>>> Regards >>>>>>>>>>>> Ravi >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:56 PM, Russell Jurney < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Talking to myself, but hopefully creating good docs. Replacing >>>>>>>>>>>>> the previous hadoop version with one I found here: >>>>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/, >>>>>>>>>>>>> 2.3.0-mr1-cdh5.1.0, >>>>>>>>>>>>> makes things get a little further. >>>>>>>>>>>>> >>>>>>>>>>>>> I can't get past some build errors, however. Has anyone done >>>>>>>>>>>>> this before me who can help? >>>>>>>>>>>>> >>>>>>>>>>>>> [ERROR] >>>>>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27] >>>>>>>>>>>>> cannot find symbol >>>>>>>>>>>>> >>>>>>>>>>>>> symbol: method valueOf(java.lang.String,int,int) >>>>>>>>>>>>> >>>>>>>>>>>>> location: class org.apache.hadoop.hbase.ServerName >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [ERROR] Failed to execute goal >>>>>>>>>>>>> org.apache.maven.plugins:maven-compiler-plugin:3.0:compile >>>>>>>>>>>>> (default-compile) on project phoenix-core: Compilation failure >>>>>>>>>>>>> >>>>>>>>>>>>> [ERROR] >>>>>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27] >>>>>>>>>>>>> cannot find symbol >>>>>>>>>>>>> >>>>>>>>>>>>> [ERROR] symbol: method valueOf(java.lang.String,int,int) >>>>>>>>>>>>> >>>>>>>>>>>>> [ERROR] location: class org.apache.hadoop.hbase.ServerName >>>>>>>>>>>>> >>>>>>>>>>>>> [ERROR] -> [Help 1] >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:41 PM, Russell Jurney < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, so it is clear to me what I have to do. I have to edit my >>>>>>>>>>>>>> pom.xml to point at CDH 5.1, which translates into: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Add the cloudera repo: >>>>>>>>>>>>>> >>>>>>>>>>>>>> <repository> >>>>>>>>>>>>>> <id>cloudera</id> >>>>>>>>>>>>>> <url> >>>>>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/ >>>>>>>>>>>>>> </url> >>>>>>>>>>>>>> </repository> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Then change the hadoop and hbase versions: >>>>>>>>>>>>>> >>>>>>>>>>>>>> <!-- Hadoop Versions --> >>>>>>>>>>>>>> >>>>>>>>>>>>>> <hbase-hadoop1.version>0.98.1-cdh5.1.0</hbase-hadoop1.version> >>>>>>>>>>>>>> >>>>>>>>>>>>>> <hbase-hadoop2.version>0.98.1-cdh5.1.0</hbase-hadoop2.version> >>>>>>>>>>>>>> <hadoop-one.version>2.3.0-cdh5.1.0</hadoop-one.version> >>>>>>>>>>>>>> <hadoop-two.version>2.3.0-cdh5.1.0</hadoop-two.version> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> However, I get this error when I build, which tells me there >>>>>>>>>>>>>> is more complex POM surgery required. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [ERROR] Failed to execute goal on project phoenix-core: Could >>>>>>>>>>>>>> not resolve dependencies for project >>>>>>>>>>>>>> org.apache.phoenix:phoenix-core:jar:4.0.0-incubating: The >>>>>>>>>>>>>> following >>>>>>>>>>>>>> artifacts could not be resolved: >>>>>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0, >>>>>>>>>>>>>> org.apache.hadoop:hadoop-test:jar:2.3.0-cdh5.1.0: Could not find >>>>>>>>>>>>>> artifact >>>>>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0 in apache >>>>>>>>>>>>>> release ( >>>>>>>>>>>>>> https://repository.apache.org/content/repositories/releases/) >>>>>>>>>>>>>> -> [Help 1] >>>>>>>>>>>>>> >>>>>>>>>>>>>> Beyond changing the versions, I do not know how to fix this. >>>>>>>>>>>>>> Can anyone help? >>>>>>>>>>>>>> >>>>>>>>>>>>>> In general, is it possible to actually handle different CDH >>>>>>>>>>>>>> versions in this project? One shouldn't have to do pom surgery >>>>>>>>>>>>>> to build >>>>>>>>>>>>>> Phoenix for the most common platform. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:15 PM, Russell Jurney < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> When I try to store data into Phoenix from Pig, I get this >>>>>>>>>>>>>>> error. I am on CDH 5.1, and Phoenix 4.0. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Anyone know how to resolve this issue? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2014-08-18 17:11:25,165 INFO >>>>>>>>>>>>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader: >>>>>>>>>>>>>>> Current split being processed >>>>>>>>>>>>>>> hdfs://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+845 >>>>>>>>>>>>>>> 2014-08-18 >>>>>>>>>>>>>>> <http://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+8452014-08-18> >>>>>>>>>>>>>>> 17:11:25,173 INFO org.apache.hadoop.mapred.TaskLogsTruncater: >>>>>>>>>>>>>>> Initializing logs' truncater with mapRetainSize=-1 and >>>>>>>>>>>>>>> reduceRetainSize=-1 >>>>>>>>>>>>>>> 2014-08-18 17:11:25,175 FATAL org.apache.hadoop.mapred.Child: >>>>>>>>>>>>>>> Error running child : java.lang.IncompatibleClassChangeError: >>>>>>>>>>>>>>> Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, >>>>>>>>>>>>>>> but class was expected >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.apache.phoenix.pig.hadoop.PhoenixOutputFormat.getRecordWriter(PhoenixOutputFormat.java:65) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:548) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:653) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) >>>>>>>>>>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >>>>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native >>>>>>>>>>>>>>> Method) >>>>>>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) >>>>>>>>>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:262) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Russell Jurney twitter.com/rjurney [email protected] >>>>>>>>>>>>>>> datasyndrome.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Russell Jurney twitter.com/rjurney [email protected] >>>>>>>>>>>>>> datasyndrome.com >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Russell Jurney twitter.com/rjurney [email protected] >>>>>>>>>>>>> datasyndrome.com >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Russell Jurney twitter.com/rjurney [email protected] >>>>>>>>>>> datasyndrome.com >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Russell Jurney twitter.com/rjurney [email protected] >>>>>>>>> datasyndrome.com >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Russell Jurney twitter.com/rjurney [email protected] >>>>>>> datasyndrome.com >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Russell Jurney twitter.com/rjurney [email protected] >>>>>> datasyndrome.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Best regards, >>>>> >>>>> - Andy >>>>> >>>>> Problems worthy of attack prove their worth by hitting back. - Piet >>>>> Hein (via Tom White) >>>>> >>>> >>>> >>>> >>>> -- >>>> Russell Jurney twitter.com/rjurney [email protected] >>>> datasyndrome.com >>>> >>> >>> >> >> >> -- >> Russell Jurney twitter.com/rjurney [email protected] datasyndrome. >> com >> > > > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) > -- Russell Jurney twitter.com/rjurney [email protected] datasyndrome.com
