> you would need to build new images anyway since you need HBase installed > and started at boot time.
Right-o. I'll be setting up a local dev environment to build fresh fedora-8 AMI's with hadoop-0.17-pre and hbase-0.1.2-pre starting in the morning. Of course, I'll probably be creating an ec2 contrib dir in hbase with hbase-init script etc. I'm tempted to try and make my image multi-versioned so that it has both the latest releases hadoop/hbase as well as pre-released versions, then make it up to user-data params to decide which will be started. The idea of course being that the running the the latest bleeding edge hbase/hadoop or latest stable could be done with the same image. -- Jim On Wed, May 7, 2008 at 5:22 PM, Chris K Wensel <[EMAIL PROTECTED]> wrote: > sorry if I wasn't clear that the new scripts and old scripts were not > compatible. thus their being in 0.17.0, not 0.16.4. > > you would need to build new images anyway since you need HBase installed > and started at boot time. > > > > > On May 7, 2008, at 3:15 PM, Jim R. Wilson wrote: > > > > I've come to the conclusion that using the contrib/ec2 scripts from > > hadoop 0.17 is incompatible with the prebuilt hadoop-0.16.1 image > > currently available in the hadoop-ec2-images bucket (ami-461df82f > > hadoop-ec2-images/hadoop-0.16.1.manifest.xml to be precise). > > > > The problem is that the user-data passed in by 0.17 has a different > > format than what is expected by the hadoop-init script packaged with > > 0.16.1. Specifically, 0.17's user -data is meant to be a comma > > delimited list of bash var settings of the form KEY=VAL, whereas > > 0.16.x seems to expect just a comma delimited list of values whose > > keys are known by their ordinal placement (that is, the first value is > > the number of instances, the second value is the name of the master > > node). > > > > So now I'm back to the idea that I'm going to have to build myself an > > ec2 AMI with hadoop 0.17 from "scratch" (using the create-instance > > scripts of course). This isn't /too/ much more work than I'd have to > > do anyway. I plan on running hbase on my cluster as well as python > > hadoop-streaming jobs which was going to require other libraries (like > > SQLAlchemy and Thrift). These items were going to necessitate > > creating my own images anyway :/ > > > > -- Jim > > > > On Wed, May 7, 2008 at 3:51 PM, Jim R. Wilson <[EMAIL PROTECTED]> > wrote: > > > > > > > > > keep the questions coming. will be glad to see HBase running on ec2, > maybe > > > > we can put your changes back into the tree. > > > > > > > > > > Thanks :) > > > > > > Just prior to this excursion, I found some things needing tweaking to > > > work with my hosting provider. Now that I'm moving to ec2, I'll be on > > > the lookout for similar issues. I'll submit any patches I end up > > > making. > > > > > > -- Jim > > > > > > > > > > > > On Wed, May 7, 2008 at 3:39 PM, Chris K Wensel <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > > > In the aforementioned EC2 wiki page, it has a configuration section > > > > > labeled "(Pre 0.17) Hadoop cluster variables (GROUP, MASTER_HOST, > > > > > NO_INSTANCES)". I had assumed that I would actually need > hadoop-0.17 > > > > > on my ec2 instances in order to forgo those instructions. It sounds > > > > > like you're telling me that all the "pre 0.17" or "0.17" specific > > > > > instructions in the wiki page refer only to the ec2 creation > scripts, > > > > > not the actual running version in the cluster, is that correct? > > > > > > > > > > > > > > > > > > > > > > > correct. > > > > > > > > the only reason these scripts are in the 0.17 branch is because they > are > > > > not backward compatible with themselves. > > > > > > > > > > > > > > > > > > > > > Also, how safe is it to run a different version of the ec2 scripts > > > > > from the actual running hadoop instance? I'm guessing it's pretty > > > > > safe since you suggested it :) > > > > > > > > > > > > > > > > > > > > > > > there are no dependencies between the EC2 scripts and Hadoop core. you > can > > > > use them with any version, as long as you build EC2 Images for the > versions > > > > of Hadoop you are after with the 'new' scripts. > > > > > > > > > > > > > > > > > > > > > Thanks again for all the help - still wrapping my mind around this > stuff. > > > > > > > > > > > > > > > > > > > > > > > keep the questions coming. will be glad to see HBase running on ec2, > maybe > > > > we can put your changes back into the tree. > > > > > > > > > > > > > > > > Chris K Wensel > > > > [EMAIL PROTECTED] > > > > http://chris.wensel.net/ > > > > http://www.cascading.org/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Chris K Wensel > [EMAIL PROTECTED] > http://chris.wensel.net/ > http://www.cascading.org/ > > > > >
