Hey Weiming, What's the output of "mount" and "strace ls /mnt/dfs"?
Thanks, Eli On Wed, Dec 16, 2009 at 10:33 PM, Weiming Lu <[email protected]> wrote: > So, Can we use one machine which is not the namenode or datanode to > mount FUSE-DFS which is start up at console instead of X Window ?. > > Yesterday, I installed the hadoop-0.18.2 on another machine, and > builded the FUSE-DFS successfully. > > Should the conf files such as hadoop-site.xml, masters and slaves in > that machine must be the same as the hadoop cluster? > I copied the hadoop-site.xml, masters and slaves from the hadoop > cluster to the new machine. > In that machine, we can do 'hadoop fs -ls /' to list the files in > hadoop cluster. > > In fuse_dfs_wrapper.sh, we set JAVA_HOME, OS_ARCH, OS_NAME, > HADOOP_HOME,CLASSPATH, and modified the command as './fuse_dfs $@' > Then, we do './fuse_dfs_wrapper.sh dfs://10.15.62.4:54310 /mnt/dfs -d' > It output: > port=54310,server=10.15.62.4 > fuse-dfs didn't recognize /mnt/dfs,-2 > fuse-dfs ignoring option -d > FUSE library version: 2.8.1 > nullpath_ok: 0 > unique: 1, opcode: INIT (26), nodeid: 0, insize: 56 > INIT: 7.9 > flags=0x0000000b > max_readahead=0x00020000 > INIT: 7.12 > flags=0x00000011 > max_readahead=0x00020000 > max_write=0x00020000 > unique: 1, success, outsize: 40 > unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56 > getattr / > unique: 2, success, outsize: 120 > > In other console, we found the /mnt/dfs is mounted, and we can 'cd dfs' > However, when we 'ls /mnt/dfs', we got the error. > ls: reading directory .: Input/output error > > The output of FUSE_DFS is: > unique: 1, success, outsize: 40 > unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56 > getattr / > unique: 2, success, outsize: 120 > unique: 3, opcode: GETATTR (3), nodeid: 1, insize: 56 > getattr / > unique: 3, success, outsize: 120 > unique: 4, opcode: OPENDIR (27), nodeid: 1, insize: 48 > unique: 4, success, outsize: 32 > unique: 5, opcode: READDIR (28), nodeid: 1, insize: 80 > readdir[0] from 0 > unique: 5, success, outsize: 200 > unique: 6, opcode: RELEASEDIR (29), nodeid: 1, insize: 64 > unique: 6, success, outsize: 16 > unique: 7, opcode: GETATTR (3), nodeid: 1, insize: 56 > getattr / > unique: 7, success, outsize: 120 > unique: 8, opcode: OPENDIR (27), nodeid: 1, insize: 48 > unique: 8, success, outsize: 32 > unique: 9, opcode: READDIR (28), nodeid: 1, insize: 80 > readdir[0] from 0 > unique: 9, success, outsize: 200 > unique: 10, opcode: RELEASEDIR (29), nodeid: 1, insize: 64 > unique: 10, success, outsize: 16 > > Can somebody help me? > Thanks. > > > > On Wed, Dec 16, 2009 at 10:06 PM, Brian Bockelman <[email protected]> > wrote: >> Sigh, I suppose this means I need to come out from under the rock I've been >> hiding under and file a JIRA. >> >> As I mentioned, the work-around is to mount FUSE-DFS when GNOME is not >> running. >> >> Brian >> >> On Dec 16, 2009, at 8:00 AM, Weiming Lu wrote: >> >>> Hi, Brian, that is the point, you are right. thanks very much. >>> We used the Ubuntu Desktop for easy programing. >>> It appears that I have to upgrade the hadoop to the latest version. >>> >>> >>> On Wed, Dec 16, 2009 at 8:39 PM, Brian Bockelman <[email protected]> >>> wrote: >>>> Hey Weiming, >>>> >>>> We've recently found a race condition in FUSE-DFS that can be triggered >>>> when you run it from a host where GNOME is also running. It causes a >>>> segfault like you report below. >>>> >>>> The patch attached works for 19.1. Another option is to shut down all the >>>> instances of GNOME on your computer, then mount. >>>> >>>> Brian >>>> >>>> >>>> >>>> >>>> On Dec 16, 2009, at 4:20 AM, Weiming Lu wrote: >>>> >>>>> Thanks very much. >>>>> We use Ubuntu 8.0.4, and the kernel is 2.6.24-16-generic. >>>>> When we run jps, it shows: >>>>> 12405 startup.jar >>>>> 5950 startup.jar >>>>> 19216 SecondaryNameNode >>>>> 25381 Jps >>>>> 19053 NameNode >>>>> 19289 JobTracker >>>>> >>>>> When I pass "-o private" for fuse-dfs just as: fuse_dfs >>>>> dfs://10.15.62.4:54310 /mnt/dfs -oprivate -d >>>>> We see the debug message: >>>>> port=54310,server=10.15.62.4 >>>>> fuse-dfs didn't recognize /mnt/dfs,-2 >>>>> fuse-dfs ignoring option -d >>>>> FUSE library version: 2.8.1 >>>>> nullpath_ok: 0 >>>>> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56 >>>>> INIT: 7.9 >>>>> flags=0x0000000b >>>>> max_readahead=0x00020000 >>>>> INIT: 7.12 >>>>> flags=0x00000011 >>>>> max_readahead=0x00020000 >>>>> max_write=0x00020000 >>>>> unique: 1, success, outsize: 40 >>>>> unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56 >>>>> getattr / >>>>> unique: 3, opcode: GETATTR (3), nodeid: 1, insize: 56 >>>>> getattr / >>>>> # >>>>> # An unexpected error has been detected by Java Runtime Environment: >>>>> # >>>>> # Internal Error (sharedRuntime.cpp:552), pid=25313, tid=1107450192 >>>>> # Error: guarantee(cb != __null,"exception happened outside >>>>> interpreter, nmethods and vtable stubs (1)") >>>>> # >>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b23 mixed mode >>>>> linux-amd64) >>>>> # An error report file with more information is saved as: >>>>> # /tmp/hs_err_pid25313.log >>>>> # >>>>> # If you would like to submit a bug report, please visit: >>>>> # http://java.sun.com/webapps/bugreport/crash.jsp >>>>> # >>>>> Aborted >>>>> >>>>> When we list the /mnt/, the message is : >>>>> ls: cannot access dfs: Transport endpoint is not connected >>>>> >>>>> Thanks. >>>>> >>>>> >>>>> On Wed, Dec 16, 2009 at 3:40 PM, Eli Collins <[email protected]> wrote: >>>>>> The "fuse-dfs didn't recognize <mntpoint>" and "fuse-dfs ignoring >>>>>> option -d" are expected, they get passed along from fuse-dfs to fuse >>>>>> (via fuse_main). >>>>>> >>>>>> Does it work if you pass -o private? Still nothing reported by dmesg? >>>>>> What does jps indicate is running?What linux distribution and kernel >>>>>> are you using? >>>>>> >>>>>> Thanks, >>>>>> Eli >>>>>> >>>>>> >>>>>> On Tue, Dec 15, 2009 at 5:54 PM, Weiming Lu <[email protected]> wrote: >>>>>>> Thanks for your reply. I have added the "-d" option when calling the >>>>>>> fuse_dfs, but I can't see any stack traces. The ONLY thing I see on >>>>>>> the console is: >>>>>>> port=54310,server=10.15.62.4 >>>>>>> fuse-dfs didn't recognize /mnt/dfs,-2 >>>>>>> fuse-dfs ignoring option -d >>>>>>> fuse-dfs ignoring option -o >>>>>>> fuse-dfs didn't recognize allow_other,-2 >>>>>>> fuse: invalid argument `allow_other' >>>>>>> >>>>>>> The option "-d" is ignored. Maybe there is something wrong in my >>>>>>> building step. >>>>>>> Can anyone help me? Thanks. >>>>>>> >>>>>>> >>>>>>> On Wed, Dec 16, 2009 at 12:49 AM, Brian Bockelman >>>>>>> <[email protected]> wrote: >>>>>>>> Hey, >>>>>>>> >>>>>>>> One thing you can do is call the fuse_dfs executable with the "-d" >>>>>>>> option. This will keep the FUSE-DFS process in the foreground of the >>>>>>>> terminal, and you will see any stack traces that occur within the >>>>>>>> Hadoop portion. Very useful for debugging. >>>>>>>> >>>>>>>> Brian >>>>>>>> >>>>>>>> On Dec 15, 2009, at 12:21 AM, Weiming Lu wrote: >>>>>>>> >>>>>>>>> Thanks very much, I have builded fuse-dfs successfully with many >>>>>>>>> tries. But there are still some problems. We can list files in hadoop >>>>>>>>> by "./hadoop fs -ls ". But after mount, no file shown in /mnt/dfs, and >>>>>>>>> there is also no message in /var/log/messages. >>>>>>>>> does anybody encounter this ? >>>>>>>>> >>>>>>>>> The hadoop we used is hadoop 0.18.2, and our platform is ubuntu >>>>>>>>> 8.0.4, amd64. >>>>>>>>> >>>>>>>>> The steps we builded are: >>>>>>>>> 1. modified src/c++/libhdfs/Makefile by >>>>>>>>> adding OS_ARCH=amd64, and JAVA_HOME >>>>>>>>> removing -32m in CPPFLAGS and LDFLAGS >>>>>>>>> 2.modified src/c++/utils/configure and src/c++/pipes/configure by >>>>>>>>> adding OS_ARCH=amd64 >>>>>>>>> making it executable by "chmod 755" >>>>>>>>> 3.ln -s /usr/lib/jvm/java-6-sun/jre/lib/amd64/server/libjvm.so >>>>>>>>> /usr/local/lib >>>>>>>>> 4.install fuse with version 2.8.1 >>>>>>>>> 5.ant compile-libhdfs -Dlibhdfs=1 >>>>>>>>> 6.ant compile-contrib -Dcompile.c++=1 -Dfusedfs=1 -Dlibhdfs=1 >>>>>>>>> -Dlibhdfs.noperms=1 >>>>>>>>> NOW we have fuse_dfs and fuse_dfs_wrapper.sh in >>>>>>>>> $HADOOP_HOME/build/contrib/fuse-dfs >>>>>>>>> >>>>>>>>> 7. add the following content in fuse_dfs_wrapper.sh >>>>>>>>> #!/bin/bash >>>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-6-sun >>>>>>>>> export OS_NAME=linux >>>>>>>>> export OS_ARCH=amd64 >>>>>>>>> export HADOOP_HOME=/home/lwm/work/hadoop >>>>>>>>> >>>>>>>>> 8. ln -s /home/lwm/work/hadoop/build/libhdfs/libhdfs.so >>>>>>>>> /usr/local/lib/libhdfs.so >>>>>>>>> >>>>>>>>> 9. modified /etc/ld.so.conf by adding /usr/local/lib >>>>>>>>> and then ldconfig >>>>>>>>> 10. sudo mkdir /mnt/dfs >>>>>>>>> 11. ./fuse_dfs_wrapper.sh dfs://10.15.62.4:54310 /mnt/dfs >>>>>>>>> we got: >>>>>>>>> port=54310,server=10.15.62.4 >>>>>>>>> fuse-dfs didn't recognize /mnt/dfs,-2 >>>>>>>>> fuse-dfs ignoring option -d >>>>>>>>> fuse-dfs ignoring option -o >>>>>>>>> fuse-dfs didn't recognize allow_other,-2 >>>>>>>>> fuse: invalid argument `allow_other' >>>>>>>>> >>>>>>>>> >>>>>>>>> 2009-12-15, "Hazem Mahmoud" <[email protected]> wrote: >>>>>>>>>> Here are some notes I took when installing fuse here on our system. >>>>>>>>>> This was >>>>>>>>>> for a hadoop 19.1 installation. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Installing Fuse/Hadoop: >>>>>>>>>> >>>>>>>>>> 2. mkdir /hdfs/client01 >>>>>>>>>> >>>>>>>>>> 4. Install Sun JDK 1.6 (must be Sun) >>>>>>>>>> >>>>>>>>>> a. RPM: jdk-6u14-ea-linux-amd64.rpm >>>>>>>>>> >>>>>>>>>> b. This DID NOT WORK: >>>>>>>>>> java version "1.6.0" >>>>>>>>>> OpenJDK Runtime Environment (build 1.6.0-b09) >>>>>>>>>> OpenJDK 64-Bit Server VM (build 1.6.0-b09, mixed mode) >>>>>>>>>> >>>>>>>>>> c. Must be this: >>>>>>>>>> java version "1.6.0_14-ea" >>>>>>>>>> Java(TM) SE Runtime Environment (build 1.6.0_14-ea-b04) >>>>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 14.0-b13, mixed mode) >>>>>>>>>> >>>>>>>>>> 5. wget >>>>>>>>>> http://newman.ultralight.org/repos/hadoop/4/x86_64/hadoop-0.19.1-7.el4.x86_64.rpm >>>>>>>>>> >>>>>>>>>> 6. wget >>>>>>>>>> http://newman.ultralight.org/repos/hadoop/4/x86_64/fuse-libs-2.7.4-8_10.el4.x86_64.rpm >>>>>>>>>> >>>>>>>>>> 7. wget >>>>>>>>>> http://dag.wieers.com/rpm/packages/fuse/fuse-2.7.3-1.el5.rf.x86_64.rpm >>>>>>>>>> OR >>>>>>>>>> wget >>>>>>>>>> http://newman.ultralight.org/repos/hadoop/4/x86_64/fuse-2.7.4-8_10.el4.x86_64.rpm >>>>>>>>>> >>>>>>>>>> 8. wget >>>>>>>>>> http://newman.ultralight.org/repos/hadoop/4/x86_64/hadoop-fuse-0.19.1-7.el4.x86_64.rpm >>>>>>>>>> >>>>>>>>>> 9. rpm –ivh hadoop-0.19.1-7.el4.x86_64.rpm >>>>>>>>>> >>>>>>>>>> 10. rpm –ivh fuse-2.7.3-1.el5.rf.x86_64.rpm >>>>>>>>>> >>>>>>>>>> 11. rpm –ivh hadoop-fuse-0.19.1-7.el4.x86_64.rpm >>>>>>>>>> >>>>>>>>>> 12. rpm –ivh fuse-libs-2.7.4-8_10.el4.x86_64.rpm >>>>>>>>>> >>>>>>>>>> 13. yum install hadoop >>>>>>>>>> >>>>>>>>>> 14. yum install hadoop-fuse fuse-libs >>>>>>>>>> >>>>>>>>>> 15. In /etc/fstab: >>>>>>>>>> >>>>>>>>>> a. hdfs# /hdfs/client01 fuse >>>>>>>>>> server=<server_hostname>,port=9000,rdbuffer=1048576,allow_other,big_writes >>>>>>>>>> 0 >>>>>>>>>> 0 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 16. Then execute command: >>>>>>>>>> >>>>>>>>>> a. mount /hdfs/client01 >>>>>>>>>> >>>>>>>>>> b. ls /hdfs/client01 >>>>>>>>>> >>>>>>>>>> 17. References: >>>>>>>>>> >>>>>>>>>> a. https://twiki.grid.iu.edu/bin/view/Storage/HadoopInstallation >>>>>>>>>> >>>>>>>>>> b. http://wiki.apache.org/hadoop/MountableHDFS >>>>>>>>>> >>>>>>>>>> 18. Error: >>>>>>>>>> >>>>>>>>>> a. After performing a “yum upgrade” on the entire system it >>>>>>>>>> broke the >>>>>>>>>> fuse installation: >>>>>>>>>> >>>>>>>>>> i. [r...@host1 ~]# >>>>>>>>>> modprobe fuse >>>>>>>>>> FATAL: Module fuse not found. >>>>>>>>>> >>>>>>>>>> ii. [r...@host1 ~]# >>>>>>>>>> mount >>>>>>>>>> /hdfs/client01 >>>>>>>>>> port=11091,server=#A##### >>>>>>>>>> fuse-dfs didn't recognize /hdfs/client01,-2 >>>>>>>>>> fuse-dfs ignoring option allow_other >>>>>>>>>> fuse: device not found, try 'modprobe fuse' first >>>>>>>>>> >>>>>>>>>> b. Solution: Reinstall newer RPM’s: >>>>>>>>>> >>>>>>>>>> i. >>>>>>>>>> fuse-2.7.4-8_10.el5.x86_64.rpm >>>>>>>>>> >>>>>>>>>> ii. >>>>>>>>>> fuse-kmdl-2.6.18-128.1.10.el5-2.7.4-8_10.el5.x86_64.rpm >>>>>>>>>> >>>>>>>>>> iii. >>>>>>>>>> fuse-libs-2.7.4-8_10.el5.x86_64.rpm >>>>>>>>>> >>>>>>>>>> iv. >>>>>>>>>> hadoop-0.19.1-7.el5.x86_64.rpm >>>>>>>>>> >>>>>>>>>> v. >>>>>>>>>> hadoop-fuse-0.19.1-8.el5.x86_64.rpm >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2009/12/13 lwm <[email protected]> >>>>>>>>>> >>>>>>>>>>> Hi all >>>>>>>>>>> I have installed hadoop with version 0.18.2, and I want to use fuse >>>>>>>>>>> in >>>>>>>>>>> hadoop. Following the src/contrib/fuse-dfs/README, I executed "ant >>>>>>>>>>> compile-contrib -Dlibhdfs=1 -Dfusedfs=1 ", an error occured. I >>>>>>>>>>> can't fix it. >>>>>>>>>>> Can anyone help me? Or is there a good install guide for me? >>>>>>>>>>> thanks. >>>>>>>>>>> When I executed "ant compile-contrib -Dcompile.c++=1 -Dfusedfs=1 >>>>>>>>>>> -Dlibhdfs.noperms=1", it was builded successfully. >>>>>>>>>>> >>>>>>>>>>> compile: >>>>>>>>>>> [echo] contrib: fuse-dfs >>>>>>>>>>> [exec] automake: Makefile.am: required file `./NEWS' not found >>>>>>>>>>> [exec] automake: Makefile.am: required file `./AUTHORS' not found >>>>>>>>>>> [exec] automake: Makefile.am: required file `./ChangeLog' not >>>>>>>>>>> found >>>>>>>>>>> [exec] src/Makefile.am:19: invalid unused variable name: >>>>>>>>>>> `AM_LDFLAGS' >>>>>>>>>>> [exec] configure: error: cannot run /bin/bash ./config.sub >>>>>>>>>>> BUILD FAILED >>>>>>>>>>> /home/wm/work/hadoop/build.xml:410: The following error occurred >>>>>>>>>>> while >>>>>>>>>>> executing this line: >>>>>>>>>>> /home/wm/work/hadoop/src/contrib/build.xml:30: The following error >>>>>>>>>>> occurred >>>>>>>>>>> while executing this line: >>>>>>>>>>> /home/wm/work/hadoop/src/contrib/fuse-dfs/build.xml:54: exec >>>>>>>>>>> returned: 1 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>>> >>>> >> >> >
