> On April 8, 2013, 9:55 p.m., Vinod Kone wrote: > > > > Ben Mahler wrote: > Hey Brenden, will you be updating this review based on the comments? > > Brenden Matthews wrote: > Yes, I'll try and finish it up this afternoon.
Cool. Make sure to rebase off the latest trunk, to get the latest hadoop fixes. > On April 8, 2013, 9:55 p.m., Vinod Kone wrote: > > hadoop/TUTORIAL.sh, line 547 > > <https://reviews.apache.org/r/10346/diff/1/?file=279031#file279031line547> > > > > Is this required by the pre-package script? If yes, please add a > > comment here. > > Brenden Matthews wrote: > In my case I use it, but it's not required. I use the hook scripts for > doing things like including custom configuration parameters (i.e., 'cp > /etc/hadoop/config/* config/') before the package is created, and copying the > package to HDFS after it's ready. Just so that I understand, is the 'pre-package' and 'post-package' scripts specific to airbnb or are they part of cloudera distributions. If its the former, I would be skeptical of adding it to the tutorial script. > On April 8, 2013, 9:55 p.m., Vinod Kone wrote: > > hadoop/TUTORIAL.sh, line 549 > > <https://reviews.apache.org/r/10346/diff/1/?file=279031#file279031line549> > > > > Are these scripts only applicable for Cloudera versions? If yes, may be > > we can just test against that, like we do everywhere else. > > Brenden Matthews wrote: > My scripts are version independent, but you could certainly write scripts > that depend on a particular version. see above. > On April 8, 2013, 9:55 p.m., Vinod Kone wrote: > > hadoop/TUTORIAL.sh, line 569 > > <https://reviews.apache.org/r/10346/diff/1/?file=279031#file279031line569> > > > > What compression does J represent? It doesn't work on my MAC (OSX 10.8). > > > > How about.. s/-cjf/-czf/ > > Brenden Matthews wrote: > This is to use the XZ format instead of gzip. Support for XZ should be > included in modern GNU systems (I don't know about others). It's nice but > not necessary. See http://tukaani.org/xz/format.html for details. Since OSX 10.8 doesn't seem to support XZ out of the box, lets stick with gzip compression. > On April 8, 2013, 9:55 p.m., Vinod Kone wrote: > > hadoop/TUTORIAL.sh, lines 581-589 > > <https://reviews.apache.org/r/10346/diff/1/?file=279031#file279031line581> > > > > I think you added this because the jobtracker was failing on your Mac? > > I would rather we find and fix that bug, than having to skip it. It just > > sets a better precedent for our code. > > > > Can you file a bug (with a stack trace and logs) on your tutorial > > output? > > Brenden Matthews wrote: > I added it because a) it was failing and b) it isn't necessary to run it > every time you rebuild the package. I'll have to check again to see if it's > still not working. But, how else would you catch any errors that you have might have introduced (for e.g. in MesosScheduler)? I presume, most often that not, the reason you are rebuilding is because of code changes. > On April 8, 2013, 9:55 p.m., Vinod Kone wrote: > > hadoop/hadoop-2.0.0-mr1-cdh4.1.2_hadoop-env.sh.patch, line 10 > > <https://reviews.apache.org/r/10346/diff/1/?file=279032#file279032line10> > > > > We definitely want to be able to run this both locally and on the > > cluster. And your change, would render local runs in a non-working state. > > > > I've sent a review out that should work for both local and remote runs. > > I will get it committed. You can base your changes off that to patch > > cdh4.2.0_hadoop-env.sh. > > Brenden Matthews wrote: > If you run from ./build/hadoop instead of ./ it should work (or at least > it does for me). I think the latter is more correct than the former, but > perhaps you have a better solution. If you pull in the latest changes from trunk, you can see how I did it. - Vinod ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10346/#review18797 ----------------------------------------------------------- On April 8, 2013, 9:21 p.m., Brenden Matthews wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/10346/ > ----------------------------------------------------------- > > (Updated April 8, 2013, 9:21 p.m.) > > > Review request for mesos. > > > Description > ------- > > - Added CDH4.2.0 support to TUTORIAL.sh > - Use .tar.xz for Hadoop dist (LZMA) > - Fixed Hadoop dist extraction in launcher > - Add pre/post packaging hooks to TUTORIAL.sh > - Don't run the jobtracker at the end by default > > > Diffs > ----- > > hadoop/Makefile.am bd8919a > hadoop/TUTORIAL.sh b0f77f3 > hadoop/hadoop-2.0.0-mr1-cdh4.1.2_hadoop-env.sh.patch c61ccec > hadoop/hadoop-2.0.0-mr1-cdh4.2.0_hadoop-env.sh.patch PRE-CREATION > hadoop/hadoop-2.0.0-mr1-cdh4.2.0_mesos.patch PRE-CREATION > src/launcher/launcher.cpp eb8d6fb > > Diff: https://reviews.apache.org/r/10346/diff/ > > > Testing > ------- > > Works in production on Hadoop at airbnb. > > > Thanks, > > Brenden Matthews > >
