See inline. On Wed, Dec 7, 2011 at 7:14 AM, Periya.Data <[email protected]> wrote:
> Thanks ! A few observations: > > - After I do export conf dir and execute "hadoop fs -ls /", I see a > different dir structure from what I see when I ssh into the machine and > execute it as root. See outputs below. > > sri@PeriyaData:~$ > sri@PeriyaData:~$ export HADOOP_CONF_DIR=/\$HOME/.whirr/HadoopCluster/ > sri@PeriyaData:~$ > sri@PeriyaData:~$ hadoop fs -ls / > Found 25 items > -rw------- 1 root root 4767328 2011-11-02 12:55 /vmlinuz > drwxr-xr-x - root root 12288 2011-12-03 10:49 /etc > dr-xr-xr-x - root root 0 2011-12-02 03:28 /proc > drwxrwxrwt - root root 4096 2011-12-05 18:07 /tmp > drwxr-xr-x - root root 4096 2011-04-25 15:50 /srv > -rw-r--r-- 1 root root 13631900 2011-11-01 22:46 /initrd.img.old > drwx------ - root root 4096 2011-11-23 22:27 /root > drwxr-xr-x - root root 4096 2011-04-21 09:50 /mnt > drwxr-xr-x - root root 4096 2011-12-02 09:01 /var > drwxr-xr-x - root root 4096 2011-10-01 19:14 /cdrom > -rw------- 1 root root 4766528 2011-10-07 14:03 /vmlinuz.old > drwxr-xr-x - root root 780 2011-12-02 16:28 /run > drwxr-xr-x - root root 4096 2011-10-23 18:27 /usr > drwx------ - root root 16384 2011-10-01 19:05 /lost+found > drwxr-xr-x - root root 4096 2011-11-22 22:26 /bin > drwxr-xr-x - root root 4096 2011-04-25 15:50 /opt > drwxr-xr-x - root root 4096 2011-10-01 19:21 /home > drwxr-xr-x - root root 4320 2011-12-02 11:29 /dev > drwxr-xr-x - root root 4096 2011-03-21 01:26 /selinux > drwxr-xr-x - root root 4096 2011-11-22 22:31 /boot > drwxr-xr-x - root root 0 2011-12-02 03:28 /sys > -rw-r--r-- 1 root root 13645361 2011-11-22 22:31 /initrd.img > drwxr-xr-x - root root 4096 2011-11-22 22:28 /lib > drwxr-xr-x - root root 4096 2011-12-03 10:49 /media > drwxr-xr-x - root root 12288 2011-11-22 22:29 /sbin > sri@PeriyaData:~$ > sri@PeriyaData:~$ This is no different from the output you get when running "ls -l /" and this is happening because Hadoop is not able to find the config file. Try: $ export HADOOP_CONF_DIR=~/.whirr/HadoopCluster/ When running "hadoop fs -ls /" you should get the same output as bellow. Note: make sure the SOCKS proxy is running. % . ~/.whirr/HadoopCluster/hadoop-proxy.sh *After SSH-ing into the master node:* > > sri@ip-10-90-131-240:~$ sudo su > root@ip-10-90-131-240:/home/users/sri# > > root@ip-10-90-131-240::/home/users/jtv# jps > 2860 Jps > 2667 JobTracker > 2088 NameNode > root@ip-10-90-131-240::/home/users/jtv# hadoop fs -ls / > Error: JAVA_HOME is not set. > root@ip-10-90-131-240::/home/users/jtv# > > *After editing (setting java home) in the .bashrc file and sourcing it , > i get the expected dir structure:* > > root@ip-10-90-131-240:/home/users/sri# hadoop fs -ls / > Found 3 items > drwxr-xr-x - hadoop supergroup 0 2011-12-05 23:09 /hadoop > drwxrwxrwx - hadoop supergroup 0 2011-12-05 23:08 /tmp > drwxrwxrwx - hadoop supergroup 0 2011-12-06 01:16 /user > root@ip-10-90-131-240:/home/users/sri# > root@ip-10-90-131-240:/home/users/sri# > > Is the above normal behavior? > It looks normal to me. I think you should be able to load data & run MR jobs as expected. Can you open an issue so that we can make sure that JAVA_HOME is exported as expected by the install script? > > Thanks, > PD/ > > > > *Questions:* >>> >>> 1. Assuming everything is fine, where does Hadoop gets installed on >>> the EC2 instance? What is the path? >>> >>> >> Run jps as root and you should see the daemons running. >> >>> >>> 1. Even if Hadoop is successfully installed on the EC2 instance, are >>> the env variables properly changed on that instance? Like, path must be >>> updated either on its .bashrc or .bash_profile ...right? >>> >>> >> Try to run "hadoop fs -ls /" as root. >> >> >
