Thanks for the explain. Also, a separate ticket is created for setJar issue.
-----Original message----- From:Edward J. Yoon <[email protected]> To:[email protected] Date:Fri, 15 Apr 2011 17:56:47 +0900 Subject:Re: Eclipse test failure for HAMA-376 P.S., It would be nice if we can separate setJar() problem into another ticket! On Fri, Apr 15, 2011 at 5:38 PM, Edward J. Yoon <[email protected]> wrote: > Again, .. > > 1. JUnit tests should be run with maven build and eclipse. * eclipse > is not a point. Please see my comment again: > > https://issues.apache.org/jira/browse/HAMA-376?focusedCommentId=13018799&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13018799 > > 2. By default ("local"), user should able to run the BSP job without > starting any daemons. It's not for unit tests. > > - "local" uses LocalBSPJobRunner. > - "localhost" means that the '127.0.0.1' server. > > In hadoop case, by default, you can run the examples using > LocalJobRunner without starting hadoop-deamons. > > ... > edward@edward-VirtualBox:~/workspace/hadoop-0.18$ ps -ef | grep java > edward 3137 1881 0 17:23 pts/0 00:00:00 grep --color=auto java > > edward@edward-VirtualBox:~/workspace/hadoop-0.18$ bin/hadoop jar > build/hadoop-0.18.4-dev-examples.jar pi 1 1 > Number of Maps = 1 Samples per Map = 1 > Wrote input for Map #0 > Starting Job > 11/04/15 17:23:28 INFO jvm.JvmMetrics: Initializing JVM Metrics with > processName=JobTracker, sessionId= > 11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : > 1 > 11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : > 1 > 11/04/15 17:23:28 INFO mapred.JobClient: Running job: job_local_0001 > 11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : > 1 > 11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : > 1 > > > > On Fri, Apr 15, 2011 at 4:50 PM, Thomas Jungblut > <[email protected]> wrote: >> I would like to have these configurations a tiny bit like Hadoop. Cause it's >> proven and easy. >> We should distinguish between a person how runs eclipse and want to develop >> a BSP, real cluster with n-slaves and a (pseudo-distributed) one machine >> cluster. >> >> Most likely when a user downloads a hama release he will setup a >> pseudo-distributed environment on his linux machine. >> This is just a shortcut version of a cluster / distributed version. Let him >> submit some examples and look whether he needs it in his production >> environment. >> If he decided to do so, he maybe wants to setup a distributed environment. >> BUT he really needs to test his own BSP and develop it. It is not easy if >> you are not familiar with the framework. >> The LocalBSPJobRunner is just the same as a LocalJobRunner in Hadoop (this >> was added for testing mapreduce stuff on your local machine AND especially >> on windows, without submitting anything to a production environment or >> running deamons). Just for stepping through the code and getting familiar >> with the framework. >> >> Nobody will have a conf directory in his own trunk when he develops for a >> company with hama. >> Most of them will setup a management utitily where configurations are stored >> and jobs are submitted, from time to time with a scheduler. >> >> Short summary of my opinion >> >> 1. User running on linux with hama deamons should receive the full >> cluster experience (just the way it is) >> 2. User running on eclipse/other IDE should receive a full debug version >> that emulates a cluster through multithreading (localjobrunner) >> >> Therefore we should not add anything strange to bsp.master.address. Just let >> this be localhost. >> >> greetings from germany >> >> 2011/4/15 Edward J. Yoon <[email protected]> >> >>> I think, many things should be fixed. >>> >>> 1. Basically the value of "bsp.master.address" is setted to >>> "localhost" In hama-default.xml: >>> >>> <name>bsp.master.address</name> >>> <value>localhost</value> >>> <description>The address of the bsp master server. Either the >>> literal string "local" or a host[:port] (where host is a name or >>> >>> If we add a LocalBSPJobRunner to Hama 0.3, I think, we should use >>> "local" as a default value. ("local" ≠ "localhost"). Description >>> strings should be fixed, too. >>> >>> 2. In eclipse, conf.get("bsp.master.address") returns null. >>> >>> String mode = conf.get("bsp.master.address"); >>> LOG.info(">>>>>> " + mode); >>> >>> It means that the config files are not loaded. If we run with advanced >>> configurations in eclipse, it'll be work but, it's really annoying. >>> >>> I think, we can set the basic configurations for JUnit tests by >>> hard-coding in MiniClusterTest constructor. >>> >>> On Thu, Apr 14, 2011 at 8:02 PM, chl501 <[email protected]> wrote: >>> > My question should be `Can't we simply use the default value, which sets >>> bsp.master.address to localhost, in hama-default.xml'? >>> > >>> > Currently when using the default value (localhost), the bspMasterAddr >>> will has the value referenced to master address/port. >>> > >>> > Resetting bsp.master.address to local will cause bspMasterAddr to null. >>> > >>> > Why do we need to reset bsp.master.address to local? >>> > >>> > Sorry for my poor English. >>> > >>> > -----Original message----- >>> > From:chl501 <[email protected]> >>> > To:hama-dev <[email protected]> >>> > Date:Thu, 14 Apr 2011 17:45:38 +0800 (CST) >>> > Subject:Re: Eclipse test failure for HAMA-376 >>> > >>> > Looks like the problem is because BSPMaster.getAddress(conf) in >>> GroomServer is trying to create socket again after BSPMaster successfully >>> occupies the ip/port (GroomServers need to wait until BSPMaster is up). >>> > >>> > in the end of NetUtil.createSocketAddr() it shows >>> > >>> > public static InetSocketAddress createSocketAddr(String target, >>> > int defaultPort) { >>> > ... >>> > return new InetSocketAddress(hostname, port); >>> > } >>> > >>> > Should we change not to use bspMasterAddr because GroomServer can >>> directly obtain bsp.master.address (ip/port) from conf/hama-site.xml? >>> > >>> > -----Original message----- >>> > From:Edward J. Yoon <[email protected]> >>> > To:[email protected],[email protected] >>> > Date:Thu, 14 Apr 2011 17:32:01 +0900 >>> > Subject:Re: Eclipse test failure for HAMA-376 >>> > >>> > Nope, "Class not found error" and "setJar()" problems are unconnected >>> > things from my report. >>> > >>> > I think, the point of miniBSPCluster problem with eclipse and maven >>> > build, is that the value of "bsp.master.address" is always null: >>> > >>> >>11/04/12 20:49:50 INFO bsp.GroomServer: groom start >>> >>>11/04/12 20:49:51 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >>> >>>11/04/12 20:49:52 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >>> >>>11/04/12 20:49:53 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >>> >>>11/04/12 20:49:54 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >>> >>>11/04/12 20:49:55 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >>> > >>> > Try to set the value of "mode" to "local" as below: >>> > >>> > public GroomServer(Configuration conf) throws IOException { >>> > .... >>> > mode = "local"; >>> > .... >>> > >>> > 2011/4/14 chl501 <[email protected]>: >>> >> It is the same issue that the jar file is missing. If it is required to >>> ask use explicitly setJar, is there any place that we can reference to the >>> build version? For instance, the BSPJob.setJar() requires to set jar path >>> e.g. /path/to/eclipse/hama-1090935/build/hama-0.3.0-test.jar where build >>> version 0.3.0 is changed according to ant's property ${version}. Otherwise, >>> we may need to parse ant file for explicitly adding this info. >>> >> >>> >> 11/04/14 14:50:13 WARN bsp.BSPJobClient: No job jar file set. User >>> classes may not be found. See BSPJob#setJar(String) or check Your jar file. >>> >> 11/04/14 14:50:13 INFO bsp.GroomServer: jobConf.getJar():null >>> >> 11/04/14 14:50:13 WARN bsp.GroomServer: Error initializing >>> attempt_201104141450_0001_000001_0: >>> >> java.lang.IllegalArgumentException: Can not create a Path from a null >>> string >>> >> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78) >>> >> at org.apache.hadoop.fs.Path.<init>(Path.java:90) >>> >> at >>> org.apache.hama.bsp.GroomServer.localizeJob(GroomServer.java:519) >>> >> at >>> org.apache.hama.bsp.GroomServer.startNewTask(GroomServer.java:451) >>> >> at >>> org.apache.hama.bsp.GroomServer.access$100(GroomServer.java:70) >>> >> at >>> org.apache.hama.bsp.GroomServer$DispatchTasksHandler.handle(GroomServer.java:144) >>> >> at >>> org.apache.hama.bsp.GroomServer$Instructor.run(GroomServer.java:192) >>> >> >>> >> >>> >>>Basically, your miniBSPCluster doesn't work with eclipse JUnit test and >>> maven build as below: >>> >> >>> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: Cleaning up the system directory >>> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: file:/tmp/hadoop/bsp/system >>> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: java.lang.NullPointerException >>> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server Responder: starting >>> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server listener on 40000: >>> starting >>> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server handler 0 on 40000: >>> starting >>> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: Starting RUNNING >>> >>>11/04/12 20:49:50 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >>> >>>11/04/12 20:49:50 INFO bsp.GroomServer: groom start >>> >>>11/04/12 20:49:51 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >>> >>>11/04/12 20:49:52 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >>> >>>11/04/12 20:49:53 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >>> >>>11/04/12 20:49:54 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >>> >>>11/04/12 20:49:55 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >>> >>>11/04/12 20:49:56 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >>> >>>^CDestroying 1 processes >>> >>>Destroying process.. >>> >>>Destroyed 1 processes >>> >>>Can you find out the reason why? >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> ChiaHung Lin >>> >> Department of Information Management >>> >> National University of Kaohsiung >>> >> Taiwan >>> >> >>> > >>> > >>> > >>> > -- >>> > Best Regards, Edward J. Yoon >>> > http://blog.udanax.org >>> > http://twitter.com/eddieyoon >>> > >>> > >>> > ChiaHung Lin >>> > Department of Information Management >>> > National University of Kaohsiung >>> > Taiwan >>> > >>> > >>> > ChiaHung Lin >>> > Department of Information Management >>> > National University of Kaohsiung >>> > Taiwan >>> > >>> >>> >>> >>> -- >>> Best Regards, Edward J. Yoon >>> http://blog.udanax.org >>> http://twitter.com/eddieyoon >>> >> >> >> >> -- >> Thomas Jungblut >> Berlin >> >> mobile: 0170-3081070 >> >> business: [email protected] >> private: [email protected] >> > > > > -- > Best Regards, Edward J. Yoon > http://blog.udanax.org > http://twitter.com/eddieyoon > -- Best Regards, Edward J. Yoon http://blog.udanax.org http://twitter.com/eddieyoon ChiaHung Lin Department of Information Management National University of Kaohsiung Taiwan
