On Sep 26, 2011, at 11:20 PM, Arun C Murthy wrote: > Roman, > > In general, we'll need to make changes upstream: > # I believe someone got HBase working. > # We made changes to Pig - rather we got help from the Pig team, particularly > Daniel. > > So, we plan to work through the rest of the stack - Hive, Oozie etc. very > soon and we'll depend on updated releases from the individual projects. >
To clarify, the changes to Pig were mainly due to it's usage of the Context Objects apis which have had changes in hadoop-0.21/hadoop-0.22. Also, we expect some pieces of the stack to change if they rely on undocumented/hidden features in MR. We are absolutely committed to ensuring end-user MR applications have full compatibility - to this end we have, long since, marked the old apis as stable & supported i.e. un-deprecated them. Arun > Arun > > On Sep 26, 2011, at 3:15 PM, Roman Shaposhnik wrote: > >> Hi Arun! >> >> Great news! Hopefuly you wouldn't mind answering some of the questions >> below... >> >> On Mon, Sep 26, 2011 at 2:07 PM, Arun C Murthy <a...@hortonworks.com> wrote: >>> NextGen MapReduce (aka MRv2, aka YARN) is coming along great: >>> # We are happy to report we've done extensive scale testing to confirm >>> stability >>> - Sort/GridMixv3 etc. at ~350nodes >>> - Scale testing with simulated clusters of ~1500 nodes >>> # Functional tests for all of MapReduce functionality >>> # Pig (0.9 & 0.9.1) working with NextGen MapReduce >> >> Is there a *released* version of Pig that compiles cleanly against .23 >> snapshots? >> Same question for Hive. >> >>> We are about to finish performance certification for both HDFS & MapReduce >>> in the next >>> couple of weeks too, after which we start integration tests with HBase, >>> Hive, Oozie etc. >> >> I'm curious -- what are these integrations tests? Can I take a look at >> them? I would >> be really nice if we can levarage those via Bigtop infrastructure. Currently >> we >> have a certain # of integration tests in Bigtop that we're running >> against a fully >> deployed stack, but it would be quite nice to have extra coverage. >> >>> Given where we are I'm confident we can have a strong hadoop-0.23.0 release >>> by late October. The current plan is to deploy to alpha clusters in >>> November. Citius, Altius, Fortius! :) >> >> Could you, please, elaborate on what will be part of that deployment? >> Which versions >> of Pig, Hive, HBase, Oozie and Mahout are you targeting? >> >> Thanks, >> Roman. >
