I think maybe I find out the reason why the bsp.jar parameter is not set even when a user creates an instance BSPJob(conf, aClass), which will set the bsp.jar parameter implicitly.
It looks like the ClassLoader retrieved is different from what is being set in RunJar. In hama, when BSPJob calls findContainingJar(), my_class.getClassLoader() returns `sun.misc.Launcher$AppClassLoader'. It is different from what is set in RunJar, which should be java.net.URLClassLoader. When executing the mapred PiEstimator example, my_class.getClassLoader() is `java.net.URLClassLoader'. So even it doesn't search/ append build/hadoop-*-dev-examples.jar, which looks like being referenced by the jar command parameter (i.e. hadoop jar <jar>), in the script, hadoop still can successfully submit job without a problem. Originally my concerns is if this is a problem in BSPMaster/ GroomServer, it may be better to solve this issue in the earlier stage. For instance, after refactoring, some execution in the patch was falsely reorganized; then it's safer not to apply the patch, which would save other developers'/ users' time. 2011/4/12 Edward J. Yoon <[email protected]>: > hum, well. > > Let's learn from other projects! > > On Tue, Apr 12, 2011 at 6:09 PM, Chia-Hung Lin <[email protected]> wrote: >> $HAMA_HOME can be removed so that we can leave user to decide the >> location of jar file. Would that be a better option? >> >> 2011/4/12 Edward J. Yoon <[email protected]>: >>>> JAR_FILE=$1 >>>> if [ -f $JAR_FILE ] ; then >>>> CLASSPATH=${CLASSPATH}:$HAMA_HOME/$JAR_FILE; >>>> fi >>> >>> User can also use the jar command like this: >>> >>> # {$HAMA_HOME}/bin/hama jar /home/edward/my.jar >>> >>> On Tue, Apr 12, 2011 at 4:59 PM, Chia-Hung Lin <[email protected]> >>> wrote: >>>> How if we add >>>> >>>> JAR_FILE=$1 >>>> if [ -f $JAR_FILE ] ; then >>>> CLASSPATH=${CLASSPATH}:$HAMA_HOME/$JAR_FILE; >>>> fi >>>> >>>> to the section where "$COMMAND" = "jar" (roughly at the line 165 in >>>> bin/hama)? >>>> >>>> Also, in build.xml, appending the following line >>>> >>>> <fileset dir="${build.dir}"> >>>> <include name="**/*.jar"/> >>>> </fileset> >>>> >>>> to the end of section where path id is `test.classpath' >>>> >>>> I give a quick test. SerializePrinting, PiEstimator and the new added >>>> test case working ok. >>>> >>>> 2011/4/12 Edward J. Yoon <[email protected]>: >>>>> Release tarball contains compiled .jar binaries. So user can download >>>>> release tarball and run without additional compiling source code. At >>>>> this point if user builds with command 'ant package', another jar >>>>> files (including duplicated classes) will be created in 'build' >>>>> directory. >>>>> >>>>> .. >>>>> /hama-0.3.0.jar >>>>> /build/hama-0.3.0.jar >>>>> >>>>> I think, one of them only added to classpath. >>>>> >>>>> On Tue, Apr 12, 2011 at 1:42 PM, Chia-Hung Lin <[email protected]> >>>>> wrote: >>>>>> I download the clean trunk from >>>>>> https://svn.apache.org/repos/asf/incubator/hama/trunk >>>>>> >>>>>> After executing `ant tar,` and the extracted tar files contain the >>>>>> following content. >>>>>> >>>>>> bin build.xml CHANGES.txt conf DISCLAIMER.txt docs >>>>>> hama-0.3.0-examples.jar hama-0.3.0.jar hama-0.3.0-test.jar lib >>>>>> LICENSE.txt NOTICE.txt pom.xml README.txt src >>>>>> >>>>>> Adding the lines that causes the problem and re-executing `ant tar,` >>>>>> the result seems the same. >>>>>> >>>>>> bin build.xml CHANGES.txt conf DISCLAIMER.txt docs >>>>>> hama-0.3.0-examples.jar hama-0.3.0.jar hama-0.3.0-test.jar lib >>>>>> LICENSE.txt NOTICE.txt pom.xml README.txt src >>>>>> >>>>>> Both of the tar files contains binary jar. Am I missing any step? Or >>>>>> how to reproduce this issue? >>>>>> >>>>>> 2011/4/11 Edward J. Yoon <[email protected]>: >>>>>>> Release tarball contains compiled .jar binaries. >>>>>>> >>>>>>> On Mon, Apr 11, 2011 at 4:26 PM, Chia-Hung Lin <[email protected]> >>>>>>> wrote: >>>>>>>> Is there any reason why this function is removed? >>>>>>>> >>>>>>>> 2011/4/11 Edward J. Yoon <[email protected]>: >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> Oh.... please re-try after run the following command: >>>>>>>>> >>>>>>>>> cp build/*.jar ./ >>>>>>>>> >>>>>>>>> It seems a problem, after below code was removed from 'bin/hama' >>>>>>>>> script. >>>>>>>>> >>>>>>>>> for f in $HAMA_HOME/build/hama-*-dev.jar; do # TODO - should be >>>>>>>>> changed before initial release >>>>>>>>> CLASSPATH=${CLASSPATH}:$f; >>>>>>>>> done >>>>>>>>> >>>>>>>>> Do you have some good idea for this problem? >>>>>>>>> >>>>>>>>> On Mon, Apr 11, 2011 at 2:13 PM, Chia-Hung Lin >>>>>>>>> <[email protected]> wrote: >>>>>>>>>> With the svn revision 1090935, I notice that it requires user >>>>>>>>>> explicitly setJar("/path/to/hama-example.jar") so that the job >>>>>>>>>> submission will work. Otherwise the GroomServer will complain >>>>>>>>>> >>>>>>>>>> org.apache.hama.bsp.GroomServer: Lost connection to BSP Master >>>>>>>>>> [server/192.168.1.123:40000]. Retrying... >>>>>>>>>> java.lang.NullPointerException ... >>>>>>>>>> >>>>>>>>>> The steps applied to execute job submission include >>>>>>>>>> >>>>>>>>>> ant clean package >>>>>>>>>> start-dfs.sh >>>>>>>>>> start-bsp.sh >>>>>>>>>> hama jar build/hama-0.3.0-examples.jar test >>>>>>>>>> >>>>>>>>>> I remember previously when executing examples e.g. SerializePrinting, >>>>>>>>>> setJar() method is not required. Am I missing any step? >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> -- >>>>>>>>>> ChiaHung Lin >>>>>>>>>> Department of Information Management >>>>>>>>>> National University of Kaohsiung >>>>>>>>>> Taiwan >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Best Regards, Edward J. Yoon >>>>>>>>> http://blog.udanax.org >>>>>>>>> http://twitter.com/eddieyoon >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> ChiaHung Lin >>>>>>>> Department of Information Management >>>>>>>> National University of Kaohsiung >>>>>>>> Taiwan >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best Regards, Edward J. Yoon >>>>>>> http://blog.udanax.org >>>>>>> http://twitter.com/eddieyoon >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> ChiaHung Lin >>>>>> Department of Information Management >>>>>> National University of Kaohsiung >>>>>> Taiwan >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Best Regards, Edward J. Yoon >>>>> http://blog.udanax.org >>>>> http://twitter.com/eddieyoon >>>>> >>>> >>>> >>>> >>>> -- >>>> ChiaHung Lin >>>> Department of Information Management >>>> National University of Kaohsiung >>>> Taiwan >>>> >>> >>> >>> >>> -- >>> Best Regards, Edward J. Yoon >>> http://blog.udanax.org >>> http://twitter.com/eddieyoon >>> >> >> >> >> -- >> ChiaHung Lin >> Department of Information Management >> National University of Kaohsiung >> Taiwan >> > > > > -- > Best Regards, Edward J. Yoon > http://blog.udanax.org > http://twitter.com/eddieyoon > -- ChiaHung Lin Department of Information Management National University of Kaohsiung Taiwan
