Sigh, I suppose this means I need to come out from under the rock I've been hiding under and file a JIRA.
As I mentioned, the work-around is to mount FUSE-DFS when GNOME is not running. Brian On Dec 16, 2009, at 8:00 AM, Weiming Lu wrote: > Hi, Brian, that is the point, you are right. thanks very much. > We used the Ubuntu Desktop for easy programing. > It appears that I have to upgrade the hadoop to the latest version. > > > On Wed, Dec 16, 2009 at 8:39 PM, Brian Bockelman <[email protected]> wrote: >> Hey Weiming, >> >> We've recently found a race condition in FUSE-DFS that can be triggered when >> you run it from a host where GNOME is also running. It causes a segfault >> like you report below. >> >> The patch attached works for 19.1. Another option is to shut down all the >> instances of GNOME on your computer, then mount. >> >> Brian >> >> >> >> >> On Dec 16, 2009, at 4:20 AM, Weiming Lu wrote: >> >>> Thanks very much. >>> We use Ubuntu 8.0.4, and the kernel is 2.6.24-16-generic. >>> When we run jps, it shows: >>> 12405 startup.jar >>> 5950 startup.jar >>> 19216 SecondaryNameNode >>> 25381 Jps >>> 19053 NameNode >>> 19289 JobTracker >>> >>> When I pass "-o private" for fuse-dfs just as: fuse_dfs >>> dfs://10.15.62.4:54310 /mnt/dfs -oprivate -d >>> We see the debug message: >>> port=54310,server=10.15.62.4 >>> fuse-dfs didn't recognize /mnt/dfs,-2 >>> fuse-dfs ignoring option -d >>> FUSE library version: 2.8.1 >>> nullpath_ok: 0 >>> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56 >>> INIT: 7.9 >>> flags=0x0000000b >>> max_readahead=0x00020000 >>> INIT: 7.12 >>> flags=0x00000011 >>> max_readahead=0x00020000 >>> max_write=0x00020000 >>> unique: 1, success, outsize: 40 >>> unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56 >>> getattr / >>> unique: 3, opcode: GETATTR (3), nodeid: 1, insize: 56 >>> getattr / >>> # >>> # An unexpected error has been detected by Java Runtime Environment: >>> # >>> # Internal Error (sharedRuntime.cpp:552), pid=25313, tid=1107450192 >>> # Error: guarantee(cb != __null,"exception happened outside >>> interpreter, nmethods and vtable stubs (1)") >>> # >>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b23 mixed mode >>> linux-amd64) >>> # An error report file with more information is saved as: >>> # /tmp/hs_err_pid25313.log >>> # >>> # If you would like to submit a bug report, please visit: >>> # http://java.sun.com/webapps/bugreport/crash.jsp >>> # >>> Aborted >>> >>> When we list the /mnt/, the message is : >>> ls: cannot access dfs: Transport endpoint is not connected >>> >>> Thanks. >>> >>> >>> On Wed, Dec 16, 2009 at 3:40 PM, Eli Collins <[email protected]> wrote: >>>> The "fuse-dfs didn't recognize <mntpoint>" and "fuse-dfs ignoring >>>> option -d" are expected, they get passed along from fuse-dfs to fuse >>>> (via fuse_main). >>>> >>>> Does it work if you pass -o private? Still nothing reported by dmesg? >>>> What does jps indicate is running?What linux distribution and kernel >>>> are you using? >>>> >>>> Thanks, >>>> Eli >>>> >>>> >>>> On Tue, Dec 15, 2009 at 5:54 PM, Weiming Lu <[email protected]> wrote: >>>>> Thanks for your reply. I have added the "-d" option when calling the >>>>> fuse_dfs, but I can't see any stack traces. The ONLY thing I see on >>>>> the console is: >>>>> port=54310,server=10.15.62.4 >>>>> fuse-dfs didn't recognize /mnt/dfs,-2 >>>>> fuse-dfs ignoring option -d >>>>> fuse-dfs ignoring option -o >>>>> fuse-dfs didn't recognize allow_other,-2 >>>>> fuse: invalid argument `allow_other' >>>>> >>>>> The option "-d" is ignored. Maybe there is something wrong in my building >>>>> step. >>>>> Can anyone help me? Thanks. >>>>> >>>>> >>>>> On Wed, Dec 16, 2009 at 12:49 AM, Brian Bockelman <[email protected]> >>>>> wrote: >>>>>> Hey, >>>>>> >>>>>> One thing you can do is call the fuse_dfs executable with the "-d" >>>>>> option. This will keep the FUSE-DFS process in the foreground of the >>>>>> terminal, and you will see any stack traces that occur within the Hadoop >>>>>> portion. Very useful for debugging. >>>>>> >>>>>> Brian >>>>>> >>>>>> On Dec 15, 2009, at 12:21 AM, Weiming Lu wrote: >>>>>> >>>>>>> Thanks very much, I have builded fuse-dfs successfully with many >>>>>>> tries. But there are still some problems. We can list files in hadoop >>>>>>> by "./hadoop fs -ls ". But after mount, no file shown in /mnt/dfs, and >>>>>>> there is also no message in /var/log/messages. >>>>>>> does anybody encounter this ? >>>>>>> >>>>>>> The hadoop we used is hadoop 0.18.2, and our platform is ubuntu 8.0.4, >>>>>>> amd64. >>>>>>> >>>>>>> The steps we builded are: >>>>>>> 1. modified src/c++/libhdfs/Makefile by >>>>>>> adding OS_ARCH=amd64, and JAVA_HOME >>>>>>> removing -32m in CPPFLAGS and LDFLAGS >>>>>>> 2.modified src/c++/utils/configure and src/c++/pipes/configure by >>>>>>> adding OS_ARCH=amd64 >>>>>>> making it executable by "chmod 755" >>>>>>> 3.ln -s /usr/lib/jvm/java-6-sun/jre/lib/amd64/server/libjvm.so >>>>>>> /usr/local/lib >>>>>>> 4.install fuse with version 2.8.1 >>>>>>> 5.ant compile-libhdfs -Dlibhdfs=1 >>>>>>> 6.ant compile-contrib -Dcompile.c++=1 -Dfusedfs=1 -Dlibhdfs=1 >>>>>>> -Dlibhdfs.noperms=1 >>>>>>> NOW we have fuse_dfs and fuse_dfs_wrapper.sh in >>>>>>> $HADOOP_HOME/build/contrib/fuse-dfs >>>>>>> >>>>>>> 7. add the following content in fuse_dfs_wrapper.sh >>>>>>> #!/bin/bash >>>>>>> export JAVA_HOME=/usr/lib/jvm/java-6-sun >>>>>>> export OS_NAME=linux >>>>>>> export OS_ARCH=amd64 >>>>>>> export HADOOP_HOME=/home/lwm/work/hadoop >>>>>>> >>>>>>> 8. ln -s /home/lwm/work/hadoop/build/libhdfs/libhdfs.so >>>>>>> /usr/local/lib/libhdfs.so >>>>>>> >>>>>>> 9. modified /etc/ld.so.conf by adding /usr/local/lib >>>>>>> and then ldconfig >>>>>>> 10. sudo mkdir /mnt/dfs >>>>>>> 11. ./fuse_dfs_wrapper.sh dfs://10.15.62.4:54310 /mnt/dfs >>>>>>> we got: >>>>>>> port=54310,server=10.15.62.4 >>>>>>> fuse-dfs didn't recognize /mnt/dfs,-2 >>>>>>> fuse-dfs ignoring option -d >>>>>>> fuse-dfs ignoring option -o >>>>>>> fuse-dfs didn't recognize allow_other,-2 >>>>>>> fuse: invalid argument `allow_other' >>>>>>> >>>>>>> >>>>>>> 2009-12-15, "Hazem Mahmoud" <[email protected]> wrote: >>>>>>>> Here are some notes I took when installing fuse here on our system. >>>>>>>> This was >>>>>>>> for a hadoop 19.1 installation. >>>>>>>> >>>>>>>> >>>>>>>> Installing Fuse/Hadoop: >>>>>>>> >>>>>>>> 2. mkdir /hdfs/client01 >>>>>>>> >>>>>>>> 4. Install Sun JDK 1.6 (must be Sun) >>>>>>>> >>>>>>>> a. RPM: jdk-6u14-ea-linux-amd64.rpm >>>>>>>> >>>>>>>> b. This DID NOT WORK: >>>>>>>> java version "1.6.0" >>>>>>>> OpenJDK Runtime Environment (build 1.6.0-b09) >>>>>>>> OpenJDK 64-Bit Server VM (build 1.6.0-b09, mixed mode) >>>>>>>> >>>>>>>> c. Must be this: >>>>>>>> java version "1.6.0_14-ea" >>>>>>>> Java(TM) SE Runtime Environment (build 1.6.0_14-ea-b04) >>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 14.0-b13, mixed mode) >>>>>>>> >>>>>>>> 5. wget >>>>>>>> http://newman.ultralight.org/repos/hadoop/4/x86_64/hadoop-0.19.1-7.el4.x86_64.rpm >>>>>>>> >>>>>>>> 6. wget >>>>>>>> http://newman.ultralight.org/repos/hadoop/4/x86_64/fuse-libs-2.7.4-8_10.el4.x86_64.rpm >>>>>>>> >>>>>>>> 7. wget >>>>>>>> http://dag.wieers.com/rpm/packages/fuse/fuse-2.7.3-1.el5.rf.x86_64.rpm >>>>>>>> OR >>>>>>>> wget >>>>>>>> http://newman.ultralight.org/repos/hadoop/4/x86_64/fuse-2.7.4-8_10.el4.x86_64.rpm >>>>>>>> >>>>>>>> 8. wget >>>>>>>> http://newman.ultralight.org/repos/hadoop/4/x86_64/hadoop-fuse-0.19.1-7.el4.x86_64.rpm >>>>>>>> >>>>>>>> 9. rpm –ivh hadoop-0.19.1-7.el4.x86_64.rpm >>>>>>>> >>>>>>>> 10. rpm –ivh fuse-2.7.3-1.el5.rf.x86_64.rpm >>>>>>>> >>>>>>>> 11. rpm –ivh hadoop-fuse-0.19.1-7.el4.x86_64.rpm >>>>>>>> >>>>>>>> 12. rpm –ivh fuse-libs-2.7.4-8_10.el4.x86_64.rpm >>>>>>>> >>>>>>>> 13. yum install hadoop >>>>>>>> >>>>>>>> 14. yum install hadoop-fuse fuse-libs >>>>>>>> >>>>>>>> 15. In /etc/fstab: >>>>>>>> >>>>>>>> a. hdfs# /hdfs/client01 fuse >>>>>>>> server=<server_hostname>,port=9000,rdbuffer=1048576,allow_other,big_writes >>>>>>>> 0 >>>>>>>> 0 >>>>>>>> >>>>>>>> >>>>>>>> 16. Then execute command: >>>>>>>> >>>>>>>> a. mount /hdfs/client01 >>>>>>>> >>>>>>>> b. ls /hdfs/client01 >>>>>>>> >>>>>>>> 17. References: >>>>>>>> >>>>>>>> a. https://twiki.grid.iu.edu/bin/view/Storage/HadoopInstallation >>>>>>>> >>>>>>>> b. http://wiki.apache.org/hadoop/MountableHDFS >>>>>>>> >>>>>>>> 18. Error: >>>>>>>> >>>>>>>> a. After performing a “yum upgrade” on the entire system it broke >>>>>>>> the >>>>>>>> fuse installation: >>>>>>>> >>>>>>>> i. [r...@host1 ~]# >>>>>>>> modprobe fuse >>>>>>>> FATAL: Module fuse not found. >>>>>>>> >>>>>>>> ii. [r...@host1 ~]# >>>>>>>> mount >>>>>>>> /hdfs/client01 >>>>>>>> port=11091,server=#A##### >>>>>>>> fuse-dfs didn't recognize /hdfs/client01,-2 >>>>>>>> fuse-dfs ignoring option allow_other >>>>>>>> fuse: device not found, try 'modprobe fuse' first >>>>>>>> >>>>>>>> b. Solution: Reinstall newer RPM’s: >>>>>>>> >>>>>>>> i. >>>>>>>> fuse-2.7.4-8_10.el5.x86_64.rpm >>>>>>>> >>>>>>>> ii. >>>>>>>> fuse-kmdl-2.6.18-128.1.10.el5-2.7.4-8_10.el5.x86_64.rpm >>>>>>>> >>>>>>>> iii. >>>>>>>> fuse-libs-2.7.4-8_10.el5.x86_64.rpm >>>>>>>> >>>>>>>> iv. >>>>>>>> hadoop-0.19.1-7.el5.x86_64.rpm >>>>>>>> >>>>>>>> v. >>>>>>>> hadoop-fuse-0.19.1-8.el5.x86_64.rpm >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2009/12/13 lwm <[email protected]> >>>>>>>> >>>>>>>>> Hi all >>>>>>>>> I have installed hadoop with version 0.18.2, and I want to use fuse in >>>>>>>>> hadoop. Following the src/contrib/fuse-dfs/README, I executed "ant >>>>>>>>> compile-contrib -Dlibhdfs=1 -Dfusedfs=1 ", an error occured. I can't >>>>>>>>> fix it. >>>>>>>>> Can anyone help me? Or is there a good install guide for me? >>>>>>>>> thanks. >>>>>>>>> When I executed "ant compile-contrib -Dcompile.c++=1 -Dfusedfs=1 >>>>>>>>> -Dlibhdfs.noperms=1", it was builded successfully. >>>>>>>>> >>>>>>>>> compile: >>>>>>>>> [echo] contrib: fuse-dfs >>>>>>>>> [exec] automake: Makefile.am: required file `./NEWS' not found >>>>>>>>> [exec] automake: Makefile.am: required file `./AUTHORS' not found >>>>>>>>> [exec] automake: Makefile.am: required file `./ChangeLog' not found >>>>>>>>> [exec] src/Makefile.am:19: invalid unused variable name: >>>>>>>>> `AM_LDFLAGS' >>>>>>>>> [exec] configure: error: cannot run /bin/bash ./config.sub >>>>>>>>> BUILD FAILED >>>>>>>>> /home/wm/work/hadoop/build.xml:410: The following error occurred while >>>>>>>>> executing this line: >>>>>>>>> /home/wm/work/hadoop/src/contrib/build.xml:30: The following error >>>>>>>>> occurred >>>>>>>>> while executing this line: >>>>>>>>> /home/wm/work/hadoop/src/contrib/fuse-dfs/build.xml:54: exec >>>>>>>>> returned: 1 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>> >>>>>> >>>>> >>>> >> >> >>
smime.p7s
Description: S/MIME cryptographic signature
