> On April 15, 2013, 1:44 a.m., Ben Mahler wrote: > > Hey Brenden, note that if you'd like to speed up the review process, you > > could split up the changes you listed in the description into separate > > reviews. > > We use a tool to help with doing chained reviews, you can find it here: > > support/post-reviews.py. > > Let me know if you'd like to try it out or need any help!
This is my first time using Review Board, so you'll have to bear with me. In the future I'll be sure to split each item into a separate commit. PS: Is there documentation anywhere on how the system is used? What do 'fixed' and 'drop' mean? > On April 15, 2013, 1:44 a.m., Ben Mahler wrote: > > src/launcher/launcher.cpp, line 315 > > <https://reviews.apache.org/r/10346/diff/2/?file=279914#file279914line315> > > > > Can you confirm 'tar xjf' and 'tar xJf' both work on OSX? > > > > If not we either need to kill them or guard them with #ifdefs. Here's the tar on Linux man page: http://linux.die.net/man/1/tar 'J' is for XZ or LZMA compression, and 'j' is for bzip2. I don't know what the status is of XZ support on Mac OS since I don't use it. I would assume that it supports gzip and bzip2. Regardless, this is for extraction on the launcher and it shouldn't break anything by also supporting XZ in addition to bzip2 and gzip. > On April 15, 2013, 1:44 a.m., Ben Mahler wrote: > > hadoop/TUTORIAL.sh, lines 547-560 > > <https://reviews.apache.org/r/10346/diff/2/?file=279908#file279908line547> > > > > I'm not sold on this, we can't have any Airbnb specific code in here. The way I see it is: Mesos already has a build system that supports automated builds with Hadoop. I don't want to have to write my own build system to work alongside this. We really want to avoid having to maintain an internal fork of Mesos. The current build system doesn't allow a way to easily automate deployment of packages after the build. We need a way to copy specific configuration files into the dist package, and then copy the final package to HDFS for use on the cluster. I think the best way to do this is have a way to run hook scripts from the tutorial. If you don't want to allow hooks into the build system, then I'd have to either a) patch the build system within our own internal fork or b) maintain a copy of the build system with our own tweaks. PS: There's no secret sauce in our hook scripts, this is all they contain: pre-package: cp /etc/hadoop/conf/* conf/ post-package cp hadoop.tar.xz /mnt/hdfs Thoughts? Perhaps you have already solved this problem another way? > On April 15, 2013, 1:44 a.m., Ben Mahler wrote: > > hadoop/TUTORIAL.sh, lines 589-601 > > <https://reviews.apache.org/r/10346/diff/2/?file=279908#file279908line589> > > > > This is the essence of the TUTORIAL, more so than being a build file, > > it is about providing users with a tutorial, so they can see what commands > > we ran to produce a working setup. So I'm skeptical of adding this. > > > > I think you should be writing custom build scripts for your particular > > Airbnb setup, rather than using the TUTORIAL.sh as a build file. Do you > > have a convincing argument otherwise? I'm using the tutorial as part of our production deployment. It works, and I want to follow the DRY principal. Perhaps it's time to graduate the script from being a tutorial to being a real part of the build system? - Brenden ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10346/#review19146 ----------------------------------------------------------- On April 9, 2013, 10:36 p.m., Brenden Matthews wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/10346/ > ----------------------------------------------------------- > > (Updated April 9, 2013, 10:36 p.m.) > > > Review request for mesos. > > > Description > ------- > > - Added CDH4.2.0 support to TUTORIAL.sh > - Fixed Hadoop dist extraction in launcher > - Add pre/post packaging hooks to TUTORIAL.sh > - Don't run the jobtracker at the end by default if ${SKIP_TESTS} > defined > > > Diffs > ----- > > hadoop/Makefile.am bd8919a > hadoop/TUTORIAL.sh f3f3d64 > hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch fb8b06b > hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 8f67ff9 > hadoop/hadoop-2.0.0-mr1-cdh4.1.2_hadoop-env.sh.patch 6d31578 > hadoop/hadoop-2.0.0-mr1-cdh4.2.0_hadoop-env.sh.patch PRE-CREATION > hadoop/hadoop-2.0.0-mr1-cdh4.2.0_mesos.patch PRE-CREATION > src/launcher/launcher.cpp bc50c5f > > Diff: https://reviews.apache.org/r/10346/diff/ > > > Testing > ------- > > Works in production on Hadoop at airbnb. > > > Thanks, > > Brenden Matthews > >
