Again, .. 1. JUnit tests should be run with maven build and eclipse. * eclipse is not a point. Please see my comment again:
https://issues.apache.org/jira/browse/HAMA-376?focusedCommentId=13018799&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13018799 2. By default ("local"), user should able to run the BSP job without starting any daemons. It's not for unit tests. - "local" uses LocalBSPJobRunner. - "localhost" means that the '127.0.0.1' server. In hadoop case, by default, you can run the examples using LocalJobRunner without starting hadoop-deamons. ... edward@edward-VirtualBox:~/workspace/hadoop-0.18$ ps -ef | grep java edward 3137 1881 0 17:23 pts/0 00:00:00 grep --color=auto java edward@edward-VirtualBox:~/workspace/hadoop-0.18$ bin/hadoop jar build/hadoop-0.18.4-dev-examples.jar pi 1 1 Number of Maps = 1 Samples per Map = 1 Wrote input for Map #0 Starting Job 11/04/15 17:23:28 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : 1 11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : 1 11/04/15 17:23:28 INFO mapred.JobClient: Running job: job_local_0001 11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : 1 11/04/15 17:23:28 INFO mapred.FileInputFormat: Total input paths to process : 1 On Fri, Apr 15, 2011 at 4:50 PM, Thomas Jungblut <[email protected]> wrote: > I would like to have these configurations a tiny bit like Hadoop. Cause it's > proven and easy. > We should distinguish between a person how runs eclipse and want to develop > a BSP, real cluster with n-slaves and a (pseudo-distributed) one machine > cluster. > > Most likely when a user downloads a hama release he will setup a > pseudo-distributed environment on his linux machine. > This is just a shortcut version of a cluster / distributed version. Let him > submit some examples and look whether he needs it in his production > environment. > If he decided to do so, he maybe wants to setup a distributed environment. > BUT he really needs to test his own BSP and develop it. It is not easy if > you are not familiar with the framework. > The LocalBSPJobRunner is just the same as a LocalJobRunner in Hadoop (this > was added for testing mapreduce stuff on your local machine AND especially > on windows, without submitting anything to a production environment or > running deamons). Just for stepping through the code and getting familiar > with the framework. > > Nobody will have a conf directory in his own trunk when he develops for a > company with hama. > Most of them will setup a management utitily where configurations are stored > and jobs are submitted, from time to time with a scheduler. > > Short summary of my opinion > > 1. User running on linux with hama deamons should receive the full > cluster experience (just the way it is) > 2. User running on eclipse/other IDE should receive a full debug version > that emulates a cluster through multithreading (localjobrunner) > > Therefore we should not add anything strange to bsp.master.address. Just let > this be localhost. > > greetings from germany > > 2011/4/15 Edward J. Yoon <[email protected]> > >> I think, many things should be fixed. >> >> 1. Basically the value of "bsp.master.address" is setted to >> "localhost" In hama-default.xml: >> >> <name>bsp.master.address</name> >> <value>localhost</value> >> <description>The address of the bsp master server. Either the >> literal string "local" or a host[:port] (where host is a name or >> >> If we add a LocalBSPJobRunner to Hama 0.3, I think, we should use >> "local" as a default value. ("local" ≠ "localhost"). Description >> strings should be fixed, too. >> >> 2. In eclipse, conf.get("bsp.master.address") returns null. >> >> String mode = conf.get("bsp.master.address"); >> LOG.info(">>>>>> " + mode); >> >> It means that the config files are not loaded. If we run with advanced >> configurations in eclipse, it'll be work but, it's really annoying. >> >> I think, we can set the basic configurations for JUnit tests by >> hard-coding in MiniClusterTest constructor. >> >> On Thu, Apr 14, 2011 at 8:02 PM, chl501 <[email protected]> wrote: >> > My question should be `Can't we simply use the default value, which sets >> bsp.master.address to localhost, in hama-default.xml'? >> > >> > Currently when using the default value (localhost), the bspMasterAddr >> will has the value referenced to master address/port. >> > >> > Resetting bsp.master.address to local will cause bspMasterAddr to null. >> > >> > Why do we need to reset bsp.master.address to local? >> > >> > Sorry for my poor English. >> > >> > -----Original message----- >> > From:chl501 <[email protected]> >> > To:hama-dev <[email protected]> >> > Date:Thu, 14 Apr 2011 17:45:38 +0800 (CST) >> > Subject:Re: Eclipse test failure for HAMA-376 >> > >> > Looks like the problem is because BSPMaster.getAddress(conf) in >> GroomServer is trying to create socket again after BSPMaster successfully >> occupies the ip/port (GroomServers need to wait until BSPMaster is up). >> > >> > in the end of NetUtil.createSocketAddr() it shows >> > >> > public static InetSocketAddress createSocketAddr(String target, >> > int defaultPort) { >> > ... >> > return new InetSocketAddress(hostname, port); >> > } >> > >> > Should we change not to use bspMasterAddr because GroomServer can >> directly obtain bsp.master.address (ip/port) from conf/hama-site.xml? >> > >> > -----Original message----- >> > From:Edward J. Yoon <[email protected]> >> > To:[email protected],[email protected] >> > Date:Thu, 14 Apr 2011 17:32:01 +0900 >> > Subject:Re: Eclipse test failure for HAMA-376 >> > >> > Nope, "Class not found error" and "setJar()" problems are unconnected >> > things from my report. >> > >> > I think, the point of miniBSPCluster problem with eclipse and maven >> > build, is that the value of "bsp.master.address" is always null: >> > >> >>11/04/12 20:49:50 INFO bsp.GroomServer: groom start >> >>>11/04/12 20:49:51 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >> >>>11/04/12 20:49:52 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >> >>>11/04/12 20:49:53 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >> >>>11/04/12 20:49:54 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >> >>>11/04/12 20:49:55 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >> > >> > Try to set the value of "mode" to "local" as below: >> > >> > public GroomServer(Configuration conf) throws IOException { >> > .... >> > mode = "local"; >> > .... >> > >> > 2011/4/14 chl501 <[email protected]>: >> >> It is the same issue that the jar file is missing. If it is required to >> ask use explicitly setJar, is there any place that we can reference to the >> build version? For instance, the BSPJob.setJar() requires to set jar path >> e.g. /path/to/eclipse/hama-1090935/build/hama-0.3.0-test.jar where build >> version 0.3.0 is changed according to ant's property ${version}. Otherwise, >> we may need to parse ant file for explicitly adding this info. >> >> >> >> 11/04/14 14:50:13 WARN bsp.BSPJobClient: No job jar file set. User >> classes may not be found. See BSPJob#setJar(String) or check Your jar file. >> >> 11/04/14 14:50:13 INFO bsp.GroomServer: jobConf.getJar():null >> >> 11/04/14 14:50:13 WARN bsp.GroomServer: Error initializing >> attempt_201104141450_0001_000001_0: >> >> java.lang.IllegalArgumentException: Can not create a Path from a null >> string >> >> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78) >> >> at org.apache.hadoop.fs.Path.<init>(Path.java:90) >> >> at >> org.apache.hama.bsp.GroomServer.localizeJob(GroomServer.java:519) >> >> at >> org.apache.hama.bsp.GroomServer.startNewTask(GroomServer.java:451) >> >> at >> org.apache.hama.bsp.GroomServer.access$100(GroomServer.java:70) >> >> at >> org.apache.hama.bsp.GroomServer$DispatchTasksHandler.handle(GroomServer.java:144) >> >> at >> org.apache.hama.bsp.GroomServer$Instructor.run(GroomServer.java:192) >> >> >> >> >> >>>Basically, your miniBSPCluster doesn't work with eclipse JUnit test and >> maven build as below: >> >> >> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: Cleaning up the system directory >> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: file:/tmp/hadoop/bsp/system >> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: java.lang.NullPointerException >> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server Responder: starting >> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server listener on 40000: >> starting >> >>>11/04/12 20:49:49 INFO ipc.Server: IPC Server handler 0 on 40000: >> starting >> >>>11/04/12 20:49:49 INFO bsp.BSPMaster: Starting RUNNING >> >>>11/04/12 20:49:50 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >> >>>11/04/12 20:49:50 INFO bsp.GroomServer: groom start >> >>>11/04/12 20:49:51 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >> >>>11/04/12 20:49:52 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >> >>>11/04/12 20:49:53 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >> >>>11/04/12 20:49:54 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >> >>>11/04/12 20:49:55 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >> >>>11/04/12 20:49:56 INFO hama.MiniBSPCluster: Waitin for GroomServer up. >> >>>^CDestroying 1 processes >> >>>Destroying process.. >> >>>Destroyed 1 processes >> >>>Can you find out the reason why? >> >> >> >> >> >> >> >> >> >> >> >> ChiaHung Lin >> >> Department of Information Management >> >> National University of Kaohsiung >> >> Taiwan >> >> >> > >> > >> > >> > -- >> > Best Regards, Edward J. Yoon >> > http://blog.udanax.org >> > http://twitter.com/eddieyoon >> > >> > >> > ChiaHung Lin >> > Department of Information Management >> > National University of Kaohsiung >> > Taiwan >> > >> > >> > ChiaHung Lin >> > Department of Information Management >> > National University of Kaohsiung >> > Taiwan >> > >> >> >> >> -- >> Best Regards, Edward J. Yoon >> http://blog.udanax.org >> http://twitter.com/eddieyoon >> > > > > -- > Thomas Jungblut > Berlin > > mobile: 0170-3081070 > > business: [email protected] > private: [email protected] > -- Best Regards, Edward J. Yoon http://blog.udanax.org http://twitter.com/eddieyoon
