Dear Wiki user, You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.
The following page has been changed by TomWhite: http://wiki.apache.org/lucene-hadoop/AmazonEC2 ------------------------------------------------------------------------------ [http://www.amazon.com/gp/browse.html?node=201590011 Amazon EC2] (Elastic Compute Cloud) is a computing service. One allocates a set of hosts, and runs ones's application on them, then, when done, de-allocates the hosts. Billing is hourly per host. Thus EC2 permits one to deploy Hadoop on a cluster without having to own and operate that cluster, but rather renting it on an hourly basis. This document assumes that you have already followed the steps in [http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-06-26/ Amazon's Getting Started Guide]. + + There are now some [#AutomatedScripts scripts] available for running Hadoop on EC2. == Concepts == @@ -222, +224 @@ Please vote for these issues in Jira if you feel this would help your project. (Anyone can create themselves a Jira account in order to vote on issues, etc.) + [[Anchor(AutomatedScripts])]] + = Automated Scripts = + + == Setting up == + * Make sure you've followed the Amazon EC2 [http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-10-01/ Getting Started Guide], sections "Setting up an Account", "Setting up the Tools" and the "Generating a Keypair" section of "Running an Instance". + * Unpack the Hadoop EC2 scripts distribution available from [http://issues.apache.org/jira/browse/HADOOP-884 HADOOP-884]. + * Edit all relevant variables in `hadoop-ec2-env.sh`. + * You need to get a Java download URL by visiting [http://java.sun.com/javase/downloads/index_jdk5.jsp here]. Make sure you get the JDK (not JRE) labelled "Linux self-extracting file". (The scripts have not been tested with Java 6 yet.) + + == Creating an image == + You only need to do this once. + * Type {{{ + ./create-hadoop-image + }}} + * Accept the Java license terms. + * The script will create a new image, then bundle, upload and register it. This may take some time. Be patient - don't assume it's crashed. + * Terminate you instance using the command given by the script. + + == Running a job on a cluster == + * Run cluster. During execution of this script you will be prompted to set up DNS (e.g. [http://www.dyndns.com/services/dns/dyndns/ DynDNS]). {{{ + ./run-hadoop-cluster + }}} + * You will then be logged into the master node where you can start your job. + * For example, to test your cluster, try {{{ + cd /usr/local/hadoop-* + bin/hadoop jar hadoop-*-examples.jar pi 10 10000000 + }}} + * You can check progress of your job at `http://MASTER_HOST:50030/`. + * When you have finished logout of the master node by typing `exit`, then shutdown the cluster with {{{ + ./terminate-hadoop-cluster + }}} +
