On Mar 22, 2010, at 3:33 AM, Dougal Ballantyne wrote: > Hi, > > We are trying to migrate from a home grown environment to a Globus > based solution for a remote compute environment. Today we create > tarballs of input data + job script and FTP to an SGE headnode. A cron > job unpacks the tarball and then submits the job script. > > We have setup a GT 4.2.1 environment with SGE. We plan to use GridFTP > + RTF for the data management and submit the jobs using GRAM4-WS. > > One issue we are having or at least trying to understand is an easy > way to setup the required directory / work area for the jobs. Today > the tarballs easily lay this out when unpacked. > > i.e. Local network > > /nfs/server1/project/RnD/input > > We make a tar from the / and then upack on the remote network to this > > /remote/nfs/server1/project/RnD/input > > The jobs output to > > /remote/nfs/server1/project/RnD/output > > Then we make a tarball starting at /nfs/... which can then be unpacked > back on the local network. > > Is it possible to do something similiar within the GRAM/RSL > environment when submitting jobs?
Yes. It's possible. GRAM 4.2.1 or older allows jobs to stage-in files from or stage-out files to a machine where jobs is submitted from. For example: &(executable=$(GLOBUS_GASS_URL)/./my_script) (parameters=par1 par2) (file_stage_in=($(GLOBUS_GASS_URL)/remote/nfs_in.tar $(HOME)/input.tar)($(GLOBUS_GASS_URL)/./input_par.txt $(HOME)/input_par.txt)) (file_stage_out=($(HOME)/output.tar $(GLOBUS_GASS_URL)/remote/nfs_out.tar) (file_clean_up=$(HOME)/nfs_out.tar) (job_type=single) The GRAM 5.0.0 or newer makes it possible to define any third-party transfer using RSL language. It means that you don't need to submit jobs from machines where you have input data or you want to transfer output data to. Fo example: &(executable=$(GLOBUS_GASS_URL)/./my_script) (parameters=par1 par2) (file_stage_in=(gsiftp://mygidftp.server.com/remote/nfs_in.tar $(HOME)/input.tar)($(GLOBUS_GASS_URL)/./input_par.txt $(HOME)/input_par.txt)) (file_stage_out=($(HOME)/output.tar gsiftp://data.server.com//remote/nfs_out.tar)) (file_clean_up=$(HOME)/nfs_out.tar) (job_type=single) The $(GLOBUS_GASS_URL) is an URL address of a machine the job is submitted from. The /./ means a working directory at the moment when the job is submitted (precisely a working directory when GASS server is started) . The $(HOME) is a home directory on a remote machine. Above jobs can be submitted the following way (nota bene the option '-w'): $ globusrun -o -w -r <GRAM_end_point> -f <rsl_file> Regards, Lukasz
