Yes, you can do the same thing using GRAM4 and an XML job description (you asked earlier for GRAM/RSL). From the WS-GRAM perspective the "utarring" requires resources, so it is a job or a part of a job. You can untar files on a client machine before you submit a job. Or if a remote machine, where WS-GRAM runs, has access to a filesystem with input tar files the untarring can be a part of a job.
Regards, Lukasz On Mar 22, 2010, at 9:27 AM, Dougal Ballantyne wrote: > Lukasz, > > Thank you for the information. We are using GRAM4-WS, do your examples > still work? > > Also it does not solve our issue, which is ensuring that a specific > directory structure exists for the jobs when then are run. We are not > running jobs on user home directories but rather in specific areas > dedicated to job storage. > > Kind regards, > > Dougal > > > On Mon, Mar 22, 2010 at 2:56 PM, Lukasz Lacinski <[email protected]> wrote: >> >> On Mar 22, 2010, at 3:33 AM, Dougal Ballantyne wrote: >> >>> Hi, >>> >>> We are trying to migrate from a home grown environment to a Globus >>> based solution for a remote compute environment. Today we create >>> tarballs of input data + job script and FTP to an SGE headnode. A cron >>> job unpacks the tarball and then submits the job script. >>> >>> We have setup a GT 4.2.1 environment with SGE. We plan to use GridFTP >>> + RTF for the data management and submit the jobs using GRAM4-WS. >>> >>> One issue we are having or at least trying to understand is an easy >>> way to setup the required directory / work area for the jobs. Today >>> the tarballs easily lay this out when unpacked. >>> >>> i.e. Local network >>> >>> /nfs/server1/project/RnD/input >>> >>> We make a tar from the / and then upack on the remote network to this >>> >>> /remote/nfs/server1/project/RnD/input >>> >>> The jobs output to >>> >>> /remote/nfs/server1/project/RnD/output >>> >>> Then we make a tarball starting at /nfs/... which can then be unpacked >>> back on the local network. >>> >>> Is it possible to do something similiar within the GRAM/RSL >>> environment when submitting jobs? >> >> >> Yes. It's possible. >> >> GRAM 4.2.1 or older allows jobs to stage-in files from or stage-out files to >> a machine where jobs is submitted from. >> For example: >> >> &(executable=$(GLOBUS_GASS_URL)/./my_script) >> (parameters=par1 par2) >> (file_stage_in=($(GLOBUS_GASS_URL)/remote/nfs_in.tar >> $(HOME)/input.tar)($(GLOBUS_GASS_URL)/./input_par.txt $(HOME)/input_par.txt)) >> (file_stage_out=($(HOME)/output.tar $(GLOBUS_GASS_URL)/remote/nfs_out.tar) >> (file_clean_up=$(HOME)/nfs_out.tar) >> (job_type=single) >> >> The GRAM 5.0.0 or newer makes it possible to define any third-party transfer >> using RSL language. It means that you don't need to submit jobs from >> machines where you have input data or you want to transfer output data to. >> Fo example: >> >> &(executable=$(GLOBUS_GASS_URL)/./my_script) >> (parameters=par1 par2) >> (file_stage_in=(gsiftp://mygidftp.server.com/remote/nfs_in.tar >> $(HOME)/input.tar)($(GLOBUS_GASS_URL)/./input_par.txt $(HOME)/input_par.txt)) >> (file_stage_out=($(HOME)/output.tar >> gsiftp://data.server.com//remote/nfs_out.tar)) >> (file_clean_up=$(HOME)/nfs_out.tar) >> (job_type=single) >> >> The $(GLOBUS_GASS_URL) is an URL address of a machine the job is submitted >> from. The /./ means a working directory at the moment when the job is >> submitted (precisely a working directory when GASS server is started) . The >> $(HOME) is a home directory on a remote machine. Above jobs can be submitted >> the following way (nota bene the option '-w'): >> $ globusrun -o -w -r <GRAM_end_point> -f <rsl_file> >> >> Regards, >> Lukasz
