Martin Feller wrote: > Dougal, > > To the best of my knowledge Gram4.x/RFT does not have such a detection > mechanism. I don't know if tools on top of Gram4 (Swift, Gridway, > others?) provide mechanisms for your use-case. > > A general note: > If you are not tied to web-services, I'd check with the Gram5 folks > if they still support what Steve described
Gram5 supports this (gass-cache). More details by Stu or Joe... >, and maybe consider going > with Gram5 (based on and improved Gram2) in the long term, because in > medium terms Gram4.x won't be supported anymore, but Gram5 will > (back to the future! :) ) > > Martin > > Steven Timm wrote: >> Dougal--I am not sure what the GT4 equivalent is for file >> stage-in but I know that the GT2 stagein does detect that the >> same file has previously been staged in, and not stage it in again. >> The cache on the far end has many hard links to the same file. >> >> Steve >> >> >> On Mon, 31 May 2010, Dougal Ballantyne wrote: >> >>> Dear GT, >>> >>> I have been working on a project for several months now researching >>> and developing a grid solution based on Globus Toolkit 4. Many thanks >>> to people who have helped me with previous issues. >>> >>> I have a slightly Off-Topic question related to how others handle a >>> particular scenario. >>> >>> We have a job generation and control application that we have added >>> support for Globus through some perl modules that call globusrun-ws. >>> When a job is generated, the program pulls from the job database the >>> associated input files and creates an XML file which lists the input >>> files in StageIn and the requested results file in StageOut. This >>> works great for a single job and jobs that all use different input >>> data. However we often have a scenario when we generate several >>> hundred jobs that all use the same input data. In our current setup we >>> would StageIn the same input file several hundred times. >>> >>> I was wondering if that was a method or known best practice within the >>> Globus Toolkit for handling this sort of scenario. I am aware that we >>> could modify the tool to stage the data first, run the jobs and then >>> remove the input file BUT that would also be a change of workflow for >>> the users. >>> >>> Your thoughts or comments greatly appreciated. >>> >>> Kind regards, >>> >>> Dougal Ballantyne >>> > >
