The issue that required you changing ports is HDFS-961. Thanks, Eli
On Fri, Apr 23, 2010 at 6:30 AM, Christian Baun <[email protected]> wrote: > Brian, > > You got it!!! :-) > It works (partly)! > > i switched to Port 9000. core-site.xml includes now: > > <property> > <name>fs.default.name</name> > > <value>hdfs://ec2-75-101-210-65.compute-1.amazonaws.com:9000</value> > <final>true</final> > </property> > > > $ hadoop fs -ls / > Found 1 items > drwxr-xr-x - hadoop supergroup 0 2010-04-23 05:18 /mnt > > $ hadoop fs -ls /mnt/ > Found 1 items > drwxr-xr-x - hadoop supergroup 0 2010-04-23 13:00 /mnt/mapred > > # ./fuse_dfs_wrapper.sh dfs://ec2-75-101-210-65.compute-1.amazonaws.com:9000 > /mnt/hdfs/ > port=9000,server=ec2-75-101-210-65.compute-1.amazonaws.com > fuse-dfs didn't recognize /mnt/hdfs/,-2 > > This tiny error message remains. > > # mount | grep fuse > fuse_dfs on /hdfs type fuse.fuse_dfs > (rw,nosuid,nodev,allow_other,default_permissions) > > # ls /mnt/hdfs/ > mnt > # mkdir /mnt/hdfs/testverzeichnis > # touch /mnt/hdfs/testdatei > # ls -l /mnt/hdfs/ > total 8 > drwxr-xr-x 3 hadoop 99 4096 2010-04-23 05:18 mnt > -rw-r--r-- 1 root 99 0 2010-04-23 13:07 testdatei > drwxr-xr-x 2 root 99 4096 2010-04-23 13:05 testverzeichnis > > In /var/log/messages there was no information about hdfs/fuse. > > Only in /var/log/user.log were these lines: > Apr 23 13:04:34 ip-10-242-231-63 fuse_dfs: mounting > dfs://ec2-75-101-210-65.compute-1.amazonaws.com:9000/ > > mkdir and touch works. But I cannot write data into files(?!). They are all > read only. > When I try to copy files "from outside" into the HDFS, only an empty file is > created and in user.log appear these error messages: > > Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: ERROR: fuse problem - could not > write all the bytes for /testordner/testfile -1!=4096fuse_impls_write.c:60 > Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: WARN: fuse problem - could not > write all the bytes for /testordner/testfile -1!=4096fuse_impls_write.c:64 > Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: ERROR: fuse problem - could not > write all the bytes for /testordner/testfile -1!=4096fuse_impls_write.c:60 > Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: WARN: fuse problem - could not > write all the bytes for /testordner/testfile -1!=4096fuse_impls_write.c:64 > Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: ERROR: dfs problem - could not > close file_handle(23486496) for /testordner/testfile fuse_impls_release.c:58 > > Weird... > > But this is a big step forward. > > Thanks a lot!!! > > Best Regards > Christian > > > Am Freitag, 23. April 2010 schrieb Brian Bockelman: >> Hm, ok, now you have me stumped. >> >> One last hunch - can you include the port information, but also switch to >> port 9000? >> >> Additionally, can you do the following: >> >> 1) In /var/log/messages and copy out the hdfs/fuse-related messages and post >> them >> 2) Using the hadoop clients do, >> hadoop fs -ls / >> >> Brian >> >> On Apr 23, 2010, at 12:33 AM, Christian Baun wrote: >> >> > Hi, >> > >> > When adding the port information inside core-site.xml, the problem remains: >> > >> > <property> >> > <name>fs.default.name</name> >> > >> > <value>hdfs://ec2-75-101-210-65.compute-1.amazonaws.com:8020</value> >> > <final>true</final> >> > </property> >> > >> > # ./fuse_dfs_wrapper.sh >> > dfs://ec2-75-101-210-65.compute-1.amazonaws.com:8020 /mnt/hdfs/ >> > port=8020,server=ec2-75-101-210-65.compute-1.amazonaws.com >> > fuse-dfs didn't recognize /mnt/hdfs/,-2 >> > >> > # ls /mnt/hdfs >> > ls: cannot access /mnt/hdfs/®1 : No such file or directory >> > >> > Best Regards, >> > Christian >> > >> > >> > Am Freitag, 23. April 2010 schrieb Christian Baun: >> >> Hi Brian, >> >> >> >> this is inside my core-site.xml >> >> >> >> <configuration> >> >> <property> >> >> <name>fs.default.name</name> >> >> >> >> <value>hdfs://ec2-75-101-210-65.compute-1.amazonaws.com/</value> >> >> <final>true</final> >> >> </property> >> >> <property> >> >> <name>hadoop.tmp.dir</name> >> >> <value>/mnt</value> >> >> <description>A base for other temporary >> >> directories.</description> >> >> </property> >> >> </configuration> >> >> >> >> Do I need to give the port here? >> >> >> >> this is inside my hdfs-site.xml >> >> >> >> <configuration> >> >> <property> >> >> <name>dfs.name.dir</name> >> >> <value>${hadoop.tmp.dir}/dfs/name</value> >> >> <final>true</final> >> >> </property> >> >> <property> >> >> <name>dfs.data.dir</name> >> >> <value>${hadoop.tmp.dir}/dfs/data</value> >> >> >> >> </property> >> >> <property> >> >> <name>fs.checkpoint.dir</name> >> >> <value>${hadoop.tmp.dir}/dfs/namesecondary</value> >> >> <final>true</final> >> >> <final>true</final> >> >> </property> >> >> </configuration> >> >> >> >> These directories do all exist >> >> >> >> # ls -l /mnt/dfs/ >> >> total 12 >> >> drwxr-xr-x 2 hadoop hadoop 4096 2010-04-23 05:08 data >> >> drwxr-xr-x 4 hadoop hadoop 4096 2010-04-23 05:17 name >> >> drwxr-xr-x 2 hadoop hadoop 4096 2010-04-23 05:08 namesecondary >> >> >> >> I don't have the config file hadoop-site.xml in /etc/... >> >> In the source directory of hadoop I have a hadoop-site.xml but with this >> >> information >> >> >> >> <!-- DO NOT PUT ANY PROPERTY IN THIS FILE. INSTEAD USE --> >> >> <!-- core-site.xml, mapred-site.xml OR hdfs-site.xml --> >> >> <!-- This empty script is to avoid picking properties from --> >> >> <!-- conf/hadoop-site.xml This would be removed once support --> >> >> <!-- for hadoop-site.xml is removed. --> >> >> >> >> Best Regards, >> >> Christian >> >> >> >> >> >> >> >> Am Freitag, 23. April 2010 schrieb Brian Bockelman: >> >>> Hey Christian, >> >>> >> >>> I've run into this before. >> >>> >> >>> Make sure that the hostname/port you give to fuse is EXACTLY the same as >> >>> listed in hadoop-site.xml. >> >>> >> >>> If these aren't the same text string (including the ":8020"), then you >> >>> get those sort of issues. >> >>> >> >>> Brian >> >>> >> >>> On Apr 22, 2010, at 5:00 AM, Christian Baun wrote: >> >>> >> >>>> Dear All, >> >>>> >> >>>> I want to test HDFS inside Amazon EC2. >> >>>> >> >>>> Two Ubuntu instances are running inside EC2. >> >>>> One server is namenode and jobtracker. The other server is the datanode. >> >>>> Cloudera (hadoop-0.20) is installed and running. >> >>>> >> >>>> Now, I want to mount HDFS. >> >>>> I tried to install contrib/fuse-dfs as described here: >> >>>> http://wiki.apache.org/hadoop/MountableHDFS >> >>>> >> >>>> The compilation worked via: >> >>>> >> >>>> # ant compile-c++-libhdfs -Dlibhdfs=1 >> >>>> # ant package -Djava5.home=/usr/lib/jvm/java-1.5.0-sun-1.5.0.06/ >> >>>> -Dforrest.home=/home/ubuntu/apache-forrest-0.8/ >> >>>> # ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1 >> >>>> >> >>>> But now, when I try to mount the filesystem: >> >>>> >> >>>> # ./fuse_dfs_wrapper.sh >> >>>> dfs://ec2-75-101-210-65.compute-1.amazonaws.com:8020 /mnt/hdfs/ -d >> >>>> port=8020,server=ec2-75-101-210-65.compute-1.amazonaws.com >> >>>> fuse-dfs didn't recognize /mnt/hdfs/,-2 >> >>>> fuse-dfs ignoring option -d >> >>>> FUSE library version: 2.8.1 >> >>>> nullpath_ok: 0 >> >>>> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56 >> >>>> INIT: 7.13 >> >>>> flags=0x0000007b >> >>>> max_readahead=0x00020000 >> >>>> INIT: 7.12 >> >>>> flags=0x00000011 >> >>>> max_readahead=0x00020000 >> >>>> max_write=0x00020000 >> >>>> unique: 1, success, outsize: 40 >> >>>> >> >>>> >> >>>> # ./fuse_dfs_wrapper.sh >> >>>> dfs://ec2-75-101-210-65.compute-1.amazonaws.com:8020 /mnt/hdfs/ >> >>>> port=8020,server=ec2-75-101-210-65.compute-1.amazonaws.com >> >>>> fuse-dfs didn't recognize /mnt/hdfs/,-2 >> >>>> >> >>>> # ls /mnt/hdfs/ >> >>>> ls: reading directory /mnt/hdfs/: Input/output error >> >>>> # ls /mnt/hdfs/ >> >>>> ls: cannot access /mnt/hdfs/o¢ : No such file or directory >> >>>> o??? >> >>>> # ls /mnt/hdfs/ >> >>>> ls: reading directory /mnt/hdfs/: Input/output error >> >>>> # ls /mnt/hdfs/ >> >>>> ls: cannot access /mnt/hdfs/`á Óÿ : No such file or directory >> >>>> `????? >> >>>> # ls /mnt/hdfs/ >> >>>> ls: reading directory /mnt/hdfs/: Input/output error >> >>>> ... >> >>>> >> >>>> >> >>>> What can I do at this point? >> >>>> >> >>>> Thanks in advance >> >>>> Christian >> >>> >> >>> >> >> >> >> >> > >> >> > > >
