[
https://issues.apache.org/jira/browse/HADOOP-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12471973
]
James P. White commented on HADOOP-952:
---------------------------------------
Hi Tom!
You wrote:
> ...
> Any problems or questions, give me a shout! (Let me know how it goes
> anyway.)
I've gotten setup on EC2 and gave your image a whirl.
The biggest problem I had was figuring out the S3_BUCKET.
I got HADOOP_VERSION wrong a couple times.
I also spent a while getting the EC2_KEYDIR and SSH_OPTS set to use my scheme.
These are the settings I wound up with:
# The Amazon S3 bucket where the Hadoop AMI you create will be stored.
S3_BUCKET=hadoop-ec2-images
# Location of EC2 keys.
# The default setting is probably OK if you set up EC2 following the Amazon
Getting Started guide.
EC2_KEYDIR=`dirname "$EC2_PRIVATE_KEY"`
# SSH options used when connecting to EC2 instances.
# Change the -i option to be the absolute path to your keypair that you set up
in the Amazon Getting Started guide.
SSH_OPTS=`echo -i "$EC2_KEYDIR"/id_rsa-gsg-keypair -o StrictHostKeyChecking=no`
# The download URL for the Sun JDK. Visit
http://java.sun.com/javase/downloads/index_jdk5.jsp and get the URL for the
"Linux self-extracting file".
JAVA_BINARY_URL=''
# The version number of the installed JDK.
JAVA_VERSION=1.5.0_11
# The EC2 group to run your cluster in.
GROUP=hadoop-cluster-group
# The version of Hadoop to install.
HADOOP_VERSION=0.11.0
I think those are somewhat better defaults. The others are much more
self-explanatory.
I also had to rerun the run-cluster code following the "Waiting before ..."
point multiple times to get the settings worked out, so I made a shortened
version (rerun-). I also made a login script (which turns out to be a good
test before doing the "Creating instances... business").
I then tried to run the pi sample job per the wiki page, but get an exception:
[EMAIL PROTECTED] ~]# cd /usr/local/hadoop-0.11.0/
[EMAIL PROTECTED] hadoop-0.11.0]# bin/hadoop jar hadoop-0.11.0-examples.jar pi
10 10000000
Number of Maps = 10 Samples per Map = 10000000
org.apache.hadoop.ipc.RemoteException: java.io.IOException:
java.lang.ArithmeticException: / by zero
at
org.apache.hadoop.dfs.FSNamesystem$Replicator.chooseTarget(FSNamesystem.java:2593)
at
org.apache.hadoop.dfs.FSNamesystem$Replicator.chooseTarget(FSNamesystem.java:2555)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:684)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:248)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
at org.apache.hadoop.ipc.Client.call(Client.java:467)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:164)
at org.apache.hadoop.dfs.$Proxy0.create(Unknown Source)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateNewBlock(DFSClient.java:1091)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:1031)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.endBlock(DFSClient.java:1255)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1345)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at
org.apache.hadoop.fs.FSDataOutputStream$Summer.close(FSDataOutputStream.java:98)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:724)
at org.apache.hadoop.examples.PiEstimator.launch(PiEstimator.java:185)
at org.apache.hadoop.examples.PiEstimator.main(PiEstimator.java:226)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:143)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:40)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
[EMAIL PROTECTED] hadoop-0.11.0]#
> Create a public (shared) Hadoop EC2 AMI
> ---------------------------------------
>
> Key: HADOOP-952
> URL: https://issues.apache.org/jira/browse/HADOOP-952
> Project: Hadoop
> Issue Type: Improvement
> Components: scripts
> Affects Versions: 0.11.0
> Reporter: Tom White
> Assigned To: Tom White
> Attachments: hadoop-952.patch
>
>
> HADOOP-884 makes it easy to run Hadoop on an EC2 cluster, but building an AMI
> (Abstract Machine Image) can take a little while. Amazon EC2 supports shared
> AMIs
> (http://developer.amazonwebservices.com/connect/entry.jspa?entryID=530&ref=featured),
> so we could provide publically available AMIs for each Hadoop release.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
