Hi,
I did spend some time on trying to run RM, NM and HS from within Eclipse,
but got partial success. Here are the steps I followed (just in case someone
is interested)
- Got the code from SVN.
- Ran the ant and mvn command for compiling Hadoop Common, HDFS and
MapReduce.
- Ran the 'mvn eclipse:eclipse' and 'ant eclipse' for the projects.
- Included the following in /etc/environment (a reboot was required for it
to take effect)
HADOOP_MAPRED_HOME="/home/praveensripati/Hadoop/trunk/mapreduce"
HADOOP_COMMON_HOME="/home/praveensripati/Hadoop/trunk/hadoop-common"
HADOOP_HDFS_HOME="/home/praveensripati/Hadoop/trunk/hdfs"
YARN_HOME="/home/praveensripati/Hadoop/hadoop-mapreduce-1.0-SNAPSHOT"
HADOOP_CONF_DIR="/home/praveensripati/Hadoop/conf"
YARN_CONF_DIR="/home/praveensripati/Hadoop/conf"
YARN_INSTALL="/home/praveensripati/Hadoop/hadoop-mapreduce-1.0-SNAPSHOT"
JAVA_HOME="/usr/lib/jvm/java-6-sun"
- Imported the java projects in Eclipse.
- In the 'Java Build Path' removed the jar dependencies in the Libraries Tab
and included the corresponding projects in the Sources Tab. Made sure that
the projects got compiled properly.
- Started the resourcemanager, nodemanager and historyserver from command
prompt using the yarn-daemon.sh command. This dumps out the java command
with the arguments. Stopped the resourcemanager, nodemanager and
historyserver again from the command propmpt.
- Ran the ResourceManager, NodeManager and JobHistoryServer from Eclipse,
passing the parameters got from the previous step.
- Created a Java project with Eclipse and copied
org.apache.hadoop.examples.RandomWriter.java and ran with the following
parameters
-Dmapreduce.job.user.name=praveensripati
-Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912
/home/praveensripati/Hadoop/trunk/hadoop-common/target/hadoop-common-0.23.0-SNAPSHOT-bin/bin/output
and got the following exceptions for each of the mapper invocation. Could
not figure out the cause of this exception.
2011-08-13 21:44:35,785 INFO mapreduce.Job (Job.java:printTaskEvents(1213))
- Task Id : attempt_1313250752016_0003_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.RandomWriter$RandomMapper
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1179)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:180)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:640)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.RandomWriter$RandomMapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1125)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1177)
... 8 more
- But was able to run the RandomWriter from command propmpt using the below
command
./hadoop jar
$HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.23.0-SNAPSHOT.jar
randomwriter -Dmapreduce.job.user.name=$USER
-Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912 -libjars
$YARN_INSTALL/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
output
Thanks,
Praveen
On Sat, Aug 13, 2011 at 10:40 PM, Praveen Sripati
<praveensrip...@gmail.com>wrote:
>
> Hi,
>
> I am able to run the RM, NM and HS from Eclipse with a couple of settings
> and run the org.apache.hadoop.examples.RandomWriter from command promptusing
> the 'hadoop jar' command. Would be nice for easier development and
> debugging of these components.
>
> But, when I run the org.apache.hadoop.examples.RandomWriter from Eclipse, I
> get the below exceptions. I created a new Java project in Eclipse and copied
> the RandomWriter.java to it. This exception is getting repeated map number
> of times, looks like the error is thrown for every map launch.
>
> Not sure what this error means, any help is appreciated.
>
> 2011-08-13 21:44:35,785 INFO mapreduce.Job
> (Job.java:printTaskEvents(1213)) - Task Id :
> attempt_1313250752016_0003_m_000001_2, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.hadoop.examples.RandomWriter$RandomMapper
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1179)
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:180)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:640)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.examples.RandomWriter$RandomMapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1125)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1177)
> ... 8 more
>
> Thanks,
> Praveen
>
> On Wed, Aug 10, 2011 at 10:56 AM, Arun C Murthy <a...@hortonworks.com>wrote:
>
>> Praveen,
>>
>> I'm not aware of any devs using eclipse to run MRv2, I haven't tried it
>> myself.
>>
>> OTOH, if you do get to we are happy to follow your lead, please feel free
>> to provide us instructions and we can update the README... thanks.
>>
>> Arun
>>
>> On Aug 9, 2011, at 7:39 PM, Praveen Sripati wrote:
>>
>> > Hi,
>> >
>> > Are there any instructions to start the ResourceManager, NodeManager,
>> > HistoryServer from Eclipse? I got the code from SVN, compiled it and ran
>> the
>> > sample program. The projects have also been exported in Eclipse. I want
>> to
>> > run the RM, NM and HS from Eclipse, so as to see the flow and fix any
>> > problems in the code.
>> >
>> > Thanks,
>> > Praveen
>>
>>
>
