You are right, thanks for pointing me. I'll do so ;) On Mon, Sep 27, 2010 at 8:02 PM, Filipe David Manana <[email protected]> wrote: > Hi Edward, > > Thanks for committing it. > > In the future, it would be nicer to commit only the changes related to > the respective ticket/patch, to make it easier for others to > follow/review the development. > > In this particular case, you committed as well changes relative to the > job paths. It would be easier for others if those changes came in a > separate commit. > > cheers > > On Mon, Sep 27, 2010 at 5:28 AM, <[email protected]> wrote: >> Author: edwardyoon >> Date: Mon Sep 27 04:28:32 2010 >> New Revision: 1001586 >> >> URL: http://svn.apache.org/viewvc?rev=1001586&view=rev >> Log: >> BSPMaster should use the bsp.master.port config property when creating its >> InetSocketAddr instance. >> >> Modified: >> incubator/hama/trunk/CHANGES.txt >> incubator/hama/trunk/conf/hama-default.xml >> incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java >> incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java >> incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java >> incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java >> >> Modified: incubator/hama/trunk/CHANGES.txt >> URL: >> http://svn.apache.org/viewvc/incubator/hama/trunk/CHANGES.txt?rev=1001586&r1=1001585&r2=1001586&view=diff >> ============================================================================== >> --- incubator/hama/trunk/CHANGES.txt (original) >> +++ incubator/hama/trunk/CHANGES.txt Mon Sep 27 04:28:32 2010 >> @@ -46,6 +46,8 @@ Trunk (unreleased changes) >> >> IMPROVEMENTS >> >> + HAMA-287: BSPMaster should use the bsp.master.port config property >> + when creating its InetSocketAddr instance (Filipe >> Manana via edwardyoon) >> HAMA-283: Removing duplicate code (Filipe Manana via edwardyoon) >> HAMA-277: Add default number of bsp task (edwardyoon) >> HAMA-273: Implement killJob() method for local job (edwardyoon) >> >> Modified: incubator/hama/trunk/conf/hama-default.xml >> URL: >> http://svn.apache.org/viewvc/incubator/hama/trunk/conf/hama-default.xml?rev=1001586&r1=1001585&r2=1001586&view=diff >> ============================================================================== >> --- incubator/hama/trunk/conf/hama-default.xml (original) >> +++ incubator/hama/trunk/conf/hama-default.xml Mon Sep 27 04:28:32 2010 >> @@ -26,7 +26,8 @@ >> <name>bsp.master.address</name> >> <value>local</value> >> <description>The address of the bsp master server. Either the >> - literal string "local" or a host:port for distributed mode >> + literal string "local" or a host[:port] (where host is a name or >> + IP address) for distributed mode. >> </description> >> </property> >> <property> >> >> Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java >> URL: >> http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java?rev=1001586&r1=1001585&r2=1001586&view=diff >> ============================================================================== >> --- incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java >> (original) >> +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPJobClient.java Mon >> Sep 27 04:28:32 2010 >> @@ -237,6 +237,8 @@ public class BSPJobClient extends Config >> Path submitJarFile = new Path(submitJobDir, "job.jar"); >> Path submitJobFile = new Path(submitJobDir, "job.xml"); >> >> + LOG.debug("BSPJobClient.submitJobDir: " + submitJobDir); >> + >> /* >> * set this user's id in job configuration, so later job files can be >> * accessed using this user's id >> >> Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java >> URL: >> http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java?rev=1001586&r1=1001585&r2=1001586&view=diff >> ============================================================================== >> --- incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java >> (original) >> +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/BSPMaster.java Mon Sep >> 27 04:28:32 2010 >> @@ -237,16 +237,10 @@ public class BSPMaster implements JobSub >> } >> >> public static InetSocketAddress getAddress(Configuration conf) { >> - String hamaMasterStr = conf.get("bsp.master.address", >> "localhost:40000"); >> - return NetUtils.createSocketAddr(hamaMasterStr); >> - } >> - >> - public int getPort() { >> - return this.conf.getInt("bsp.master.port", 0); >> - } >> + String hamaMasterStr = conf.get("bsp.master.address", "localhost"); >> + int defaultPort = conf.getInt("bsp.master.port", 40000); >> >> - public Configuration getConf() { >> - return this.conf; >> + return NetUtils.createSocketAddr(hamaMasterStr, defaultPort); >> } >> >> private static SimpleDateFormat getDateFormat() { >> >> Modified: incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java >> URL: >> http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java?rev=1001586&r1=1001585&r2=1001586&view=diff >> ============================================================================== >> --- incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java >> (original) >> +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/GroomServer.java Mon >> Sep 27 04:28:32 2010 >> @@ -391,23 +391,32 @@ public class GroomServer implements Runn >> this.task = task; >> } >> >> + static final String SUBDIR = "groomServer"; >> + >> public void launchTask() { >> // until job is completed, don't accept new task >> acceptNewTasks = false; >> >> try { >> // TODO: need to move this code to TaskRunner >> - Path localJobFile = new >> Path(task.getJobFile().replace(systemFS.getUri().toString(), "")); >> - String localJarFile = localJobFile.toString().replace(".xml", >> ".jar"); >> + task.getJobFile(); >> + conf.addResource(task.getJobFile()); >> + BSPJob defaultJobConf = new BSPJob((HamaConfiguration) conf); >> + >> + Path localJobFile = >> + >> defaultJobConf.getLocalPath(SUBDIR+"/"+task.getTaskID()+"/"+"job.xml"); >> + Path localJarFile = >> + >> defaultJobConf.getLocalPath(SUBDIR+"/"+task.getTaskID()+"/"+"job.jar"); >> + >> LOG.debug("localJobFile: "+ localJobFile); >> >> systemFS.copyToLocalFile(new Path(task.getJobFile()), localJobFile); >> - systemFS.copyToLocalFile(new Path(task.getJobFile().replace(".xml", >> ".jar")), new Path(localJarFile)); >> + systemFS.copyToLocalFile(new Path(task.getJobFile().replace(".xml", >> ".jar")), localJarFile); >> >> HamaConfiguration conf = new HamaConfiguration(); >> conf.addResource(localJobFile); >> BSPJob jobConf = new BSPJob(conf, task.getJobID()); >> - jobConf.setJar(localJarFile); >> + jobConf.setJar(localJarFile.toString()); >> >> BSP bsp = (BSP) ReflectionUtils.newInstance(jobConf.getBspClass(), >> conf); >> bsp.setPeer(bspPeer); >> >> Modified: >> incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java >> URL: >> http://svn.apache.org/viewvc/incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java?rev=1001586&r1=1001585&r2=1001586&view=diff >> ============================================================================== >> --- incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java >> (original) >> +++ incubator/hama/trunk/src/java/org/apache/hama/bsp/JobInProgress.java Mon >> Sep 27 04:28:32 2010 >> @@ -76,6 +76,10 @@ class JobInProgress { >> + ".xml"); >> this.localJarFile = master.getLocalPath(BSPMaster.SUBDIR + "/" + jobId >> + ".jar"); >> + >> + LOG.debug("JobInProgress.localJobFile: " + this.localJobFile); >> + LOG.debug("JobInProgress.localJarFile: " + this.localJarFile); >> + >> Path jobDir = master.getSystemDirectoryForJob(jobId); >> FileSystem fs = jobDir.getFileSystem(conf); >> jobFile = new Path(jobDir, "job.xml"); >> >> >> > > > > -- > Filipe David Manana, > [email protected], [email protected] > > "Reasonable men adapt themselves to the world. > Unreasonable men adapt the world to themselves. > That's why all progress depends on unreasonable men." >
-- Best Regards, Edward J. Yoon [email protected] http://blog.udanax.org
