Hi Brian, The error is found and the filesystem works now!
The error logs of "hadoop fs -put" helped a lot. I tried to copy a small file: # ls -l /tmp/neue_datei.txt -rw-r--r-- 1 root root 5 2010-04-23 14:08 /tmp/neue_datei.txt # hadoop fs -put /tmp/neue_datei.txt /hdfs/ 10/04/23 14:09:02 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hdfs could only be replicated to 0 nodes, instead of 1 ... This told me to look into http://ec2-75-101-210-65.compute-1.amazonaws.com:50070 and http://ec2-75-101-210-65.compute-1.amazonaws.com:50030 and the logs there. The result: 0 Nodes. The namenode and jobtracker logs are full of error messages. => reboot... Now I have again 1 Live Node. # ./fuse_dfs_wrapper.sh dfs://ec2-75-101-210-65.compute-1.amazonaws.com:9000 /hdfs/ port=9000,server=ec2-75-101-210-65.compute-1.amazonaws.com fuse-dfs didn't recognize /hdfs/,-2 port=9000,server=ec2-75-101-210-65.compute-1.amazonaws.com # df | grep hdfs fuse_dfs 433455104 0 433455104 0% /hdfs But writing inside /hdfs works only for the user "hadoop" $ echo test > /hdfs/testfile $ cat /hdfs/testfile test When I try as root, it doesn't work (bash: testfile: Input/output error) and I have this error messages inside the logfile of the namenode: 2010-04-23 14:58:13,306 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9000, call create(/testfile.txt, rwxr-xr-x, DFSClient_-1937513051, true, 3, 67108864) from 10.242.231.63:46719: error: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="":hadoop:supergroup:rwxr-xr-x org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="":hadoop:supergroup:rwxr-xr-x ... But it works now. Thanks a lot for your help! Best Regards Christian Am Freitag, 23. April 2010 schrieb Brian Bockelman: > Hey Christian, > > Glad to hear things are beginning to click. Can you upload the things you > learned into the wiki? In our internal user docs, we have big bold letters > saying to watch out for this issue. > > As far as your writing issues - can you write using "hadoop fs -put"? The > nice thing about the built-in utilities is that it will give you better > terminal feedback. > > Alternately, I find myself mounting things in debug mode to see the Hadoop > issues printed out to the terminal. > > Brian > > On Apr 23, 2010, at 8:30 AM, Christian Baun wrote: > > > Brian, > > > > You got it!!! :-) > > It works (partly)! > > > > i switched to Port 9000. core-site.xml includes now: > > > > <property> > > <name>fs.default.name</name> > > > > <value>hdfs://ec2-75-101-210-65.compute-1.amazonaws.com:9000</value> > > <final>true</final> > > </property> > > > > > > $ hadoop fs -ls / > > Found 1 items > > drwxr-xr-x - hadoop supergroup 0 2010-04-23 05:18 /mnt > > > > $ hadoop fs -ls /mnt/ > > Found 1 items > > drwxr-xr-x - hadoop supergroup 0 2010-04-23 13:00 /mnt/mapred > > > > # ./fuse_dfs_wrapper.sh > > dfs://ec2-75-101-210-65.compute-1.amazonaws.com:9000 /mnt/hdfs/ > > port=9000,server=ec2-75-101-210-65.compute-1.amazonaws.com > > fuse-dfs didn't recognize /mnt/hdfs/,-2 > > > > This tiny error message remains. > > > > # mount | grep fuse > > fuse_dfs on /hdfs type fuse.fuse_dfs > > (rw,nosuid,nodev,allow_other,default_permissions) > > > > # ls /mnt/hdfs/ > > mnt > > # mkdir /mnt/hdfs/testverzeichnis > > # touch /mnt/hdfs/testdatei > > # ls -l /mnt/hdfs/ > > total 8 > > drwxr-xr-x 3 hadoop 99 4096 2010-04-23 05:18 mnt > > -rw-r--r-- 1 root 99 0 2010-04-23 13:07 testdatei > > drwxr-xr-x 2 root 99 4096 2010-04-23 13:05 testverzeichnis > > > > In /var/log/messages there was no information about hdfs/fuse. > > > > Only in /var/log/user.log were these lines: > > Apr 23 13:04:34 ip-10-242-231-63 fuse_dfs: mounting > > dfs://ec2-75-101-210-65.compute-1.amazonaws.com:9000/ > > > > mkdir and touch works. But I cannot write data into files(?!). They are all > > read only. > > When I try to copy files "from outside" into the HDFS, only an empty file > > is created and in user.log appear these error messages: > > > > Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: ERROR: fuse problem - could not > > write all the bytes for /testordner/testfile -1!=4096fuse_impls_write.c:60 > > Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: WARN: fuse problem - could not > > write all the bytes for /testordner/testfile -1!=4096fuse_impls_write.c:64 > > Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: ERROR: fuse problem - could not > > write all the bytes for /testordner/testfile -1!=4096fuse_impls_write.c:60 > > Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: WARN: fuse problem - could not > > write all the bytes for /testordner/testfile -1!=4096fuse_impls_write.c:64 > > Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: ERROR: dfs problem - could not > > close file_handle(23486496) for /testordner/testfile fuse_impls_release.c:58 > > > > Weird... > > > > But this is a big step forward. > > > > Thanks a lot!!! > > > > Best Regards > > Christian > > > > > > Am Freitag, 23. April 2010 schrieb Brian Bockelman: > >> Hm, ok, now you have me stumped. > >> > >> One last hunch - can you include the port information, but also switch to > >> port 9000? > >> > >> Additionally, can you do the following: > >> > >> 1) In /var/log/messages and copy out the hdfs/fuse-related messages and > >> post them > >> 2) Using the hadoop clients do, > >> hadoop fs -ls / > >> > >> Brian > >> > >> On Apr 23, 2010, at 12:33 AM, Christian Baun wrote: > >> > >>> Hi, > >>> > >>> When adding the port information inside core-site.xml, the problem > >>> remains: > >>> > >>> <property> > >>> <name>fs.default.name</name> > >>> > >>> <value>hdfs://ec2-75-101-210-65.compute-1.amazonaws.com:8020</value> > >>> <final>true</final> > >>> </property> > >>> > >>> # ./fuse_dfs_wrapper.sh > >>> dfs://ec2-75-101-210-65.compute-1.amazonaws.com:8020 /mnt/hdfs/ > >>> port=8020,server=ec2-75-101-210-65.compute-1.amazonaws.com > >>> fuse-dfs didn't recognize /mnt/hdfs/,-2 > >>> > >>> # ls /mnt/hdfs > >>> ls: cannot access /mnt/hdfs/®1: No such file or directory > >>> > >>> Best Regards, > >>> Christian > >>> > >>> > >>> Am Freitag, 23. April 2010 schrieb Christian Baun: > >>>> Hi Brian, > >>>> > >>>> this is inside my core-site.xml > >>>> > >>>> <configuration> > >>>> <property> > >>>> <name>fs.default.name</name> > >>>> <value>hdfs://ec2-75-101-210-65.compute-1.amazonaws.com/</value> > >>>> <final>true</final> > >>>> </property> > >>>> <property> > >>>> <name>hadoop.tmp.dir</name> > >>>> <value>/mnt</value> > >>>> <description>A base for other temporary > >>>> directories.</description> > >>>> </property> > >>>> </configuration> > >>>> > >>>> Do I need to give the port here? > >>>> > >>>> this is inside my hdfs-site.xml > >>>> > >>>> <configuration> > >>>> <property> > >>>> <name>dfs.name.dir</name> > >>>> <value>${hadoop.tmp.dir}/dfs/name</value> > >>>> <final>true</final> > >>>> </property> > >>>> <property> > >>>> <name>dfs.data.dir</name> > >>>> <value>${hadoop.tmp.dir}/dfs/data</value> > >>>> > >>>> </property> > >>>> <property> > >>>> <name>fs.checkpoint.dir</name> > >>>> <value>${hadoop.tmp.dir}/dfs/namesecondary</value> > >>>> <final>true</final> > >>>> <final>true</final> > >>>> </property> > >>>> </configuration> > >>>> > >>>> These directories do all exist > >>>> > >>>> # ls -l /mnt/dfs/ > >>>> total 12 > >>>> drwxr-xr-x 2 hadoop hadoop 4096 2010-04-23 05:08 data > >>>> drwxr-xr-x 4 hadoop hadoop 4096 2010-04-23 05:17 name > >>>> drwxr-xr-x 2 hadoop hadoop 4096 2010-04-23 05:08 namesecondary > >>>> > >>>> I don't have the config file hadoop-site.xml in /etc/... > >>>> In the source directory of hadoop I have a hadoop-site.xml but with this > >>>> information > >>>> > >>>> <!-- DO NOT PUT ANY PROPERTY IN THIS FILE. INSTEAD USE --> > >>>> <!-- core-site.xml, mapred-site.xml OR hdfs-site.xml --> > >>>> <!-- This empty script is to avoid picking properties from --> > >>>> <!-- conf/hadoop-site.xml This would be removed once support --> > >>>> <!-- for hadoop-site.xml is removed. --> > >>>> > >>>> Best Regards, > >>>> Christian > >>>> > >>>> > >>>> > >>>> Am Freitag, 23. April 2010 schrieb Brian Bockelman: > >>>>> Hey Christian, > >>>>> > >>>>> I've run into this before. > >>>>> > >>>>> Make sure that the hostname/port you give to fuse is EXACTLY the same > >>>>> as listed in hadoop-site.xml. > >>>>> > >>>>> If these aren't the same text string (including the ":8020"), then you > >>>>> get those sort of issues. > >>>>> > >>>>> Brian > >>>>> > >>>>> On Apr 22, 2010, at 5:00 AM, Christian Baun wrote: > >>>>> > >>>>>> Dear All, > >>>>>> > >>>>>> I want to test HDFS inside Amazon EC2. > >>>>>> > >>>>>> Two Ubuntu instances are running inside EC2. > >>>>>> One server is namenode and jobtracker. The other server is the > >>>>>> datanode. > >>>>>> Cloudera (hadoop-0.20) is installed and running. > >>>>>> > >>>>>> Now, I want to mount HDFS. > >>>>>> I tried to install contrib/fuse-dfs as described here: > >>>>>> http://wiki.apache.org/hadoop/MountableHDFS > >>>>>> > >>>>>> The compilation worked via: > >>>>>> > >>>>>> # ant compile-c++-libhdfs -Dlibhdfs=1 > >>>>>> # ant package -Djava5.home=/usr/lib/jvm/java-1.5.0-sun-1.5.0.06/ > >>>>>> -Dforrest.home=/home/ubuntu/apache-forrest-0.8/ > >>>>>> # ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1 > >>>>>> > >>>>>> But now, when I try to mount the filesystem: > >>>>>> > >>>>>> # ./fuse_dfs_wrapper.sh > >>>>>> dfs://ec2-75-101-210-65.compute-1.amazonaws.com:8020 /mnt/hdfs/ -d > >>>>>> port=8020,server=ec2-75-101-210-65.compute-1.amazonaws.com > >>>>>> fuse-dfs didn't recognize /mnt/hdfs/,-2 > >>>>>> fuse-dfs ignoring option -d > >>>>>> FUSE library version: 2.8.1 > >>>>>> nullpath_ok: 0 > >>>>>> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56 > >>>>>> INIT: 7.13 > >>>>>> flags=0x0000007b > >>>>>> max_readahead=0x00020000 > >>>>>> INIT: 7.12 > >>>>>> flags=0x00000011 > >>>>>> max_readahead=0x00020000 > >>>>>> max_write=0x00020000 > >>>>>> unique: 1, success, outsize: 40 > >>>>>> > >>>>>> > >>>>>> # ./fuse_dfs_wrapper.sh > >>>>>> dfs://ec2-75-101-210-65.compute-1.amazonaws.com:8020 /mnt/hdfs/ > >>>>>> port=8020,server=ec2-75-101-210-65.compute-1.amazonaws.com > >>>>>> fuse-dfs didn't recognize /mnt/hdfs/,-2 > >>>>>> > >>>>>> # ls /mnt/hdfs/ > >>>>>> ls: reading directory /mnt/hdfs/: Input/output error > >>>>>> # ls /mnt/hdfs/ > >>>>>> ls: cannot access /mnt/hdfs/o¢: No such file or directory > >>>>>> o??? > >>>>>> # ls /mnt/hdfs/ > >>>>>> ls: reading directory /mnt/hdfs/: Input/output error > >>>>>> # ls /mnt/hdfs/ > >>>>>> ls: cannot access /mnt/hdfs/`áÓÿ: No such file or directory > >>>>>> `????? > >>>>>> # ls /mnt/hdfs/ > >>>>>> ls: reading directory /mnt/hdfs/: Input/output error > >>>>>> ... > >>>>>> > >>>>>> > >>>>>> What can I do at this point? > >>>>>> > >>>>>> Thanks in advance > >>>>>> Christian > >>>>> > >>>>> > >>>> > >>>> > >>> > >> > >> > > > >
