Hi,
I recently posted this message on rft-user, but seeing as that list doesn't
get much traffic, I hope no one minds if I try again here.
I'm trying to implement a file caching scheme with GRAM/RFT/GridFTP, such
that when GRAM jobs are submitted to remote resources, input files end up in
a cache of sorts on the remote resource. This is all in an effort to cut
down on unnecessary file duplication between jobs that are submitted.
The structure of the cache might look like this:
${GLOBUS_SCRATCH_DIR}/cache/md5sum_file1/md5sum
${GLOBUS_SCRATCH_DIR}/cache/md5sum_file1/foo ->
${GLOBUS_SCRATCH_DIR}/cache/md5sum_file1/md5sum
${GLOBUS_SCRATCH_DIR}/cache/md5sum_file1/bar ->
${GLOBUS_SCRATCH_DIR}/cache/md5sum_file1/md5sum
${GLOBUS_SCRATCH_DIR}/cache/md5sum_file1/baz ->
${GLOBUS_SCRATCH_DIR}/cache/md5sum_file1/md5sum
${GLOBUS_SCRATCH_DIR}/cache/md5sum_file2/md5sum
${GLOBUS_SCRATCH_DIR}/cache/md5sum_file2/foo ->
${GLOBUS_SCRATCH_DIR}/cache/md5sum_file2/md5sum
${GLOBUS_SCRATCH_DIR}/cache/md5sum_file2/bar ->
${GLOBUS_SCRATCH_DIR}/cache/md5sum_file2/md5sum
${GLOBUS_SCRATCH_DIR}/cache/md5sum_file2/baz ->
${GLOBUS_SCRATCH_DIR}/cache/md5sum_file2/md5sum
In this example, you can imagine that md5sum, md5sum_file1, and md5sum_file2
are all actual md5sums.
There are really only three cases that need to be handled:
If file exists in the cache with the same name
do nothing
If file exists in the cache with a different name
create a symlink in the cache
Else
upload file and create symlink
Ideally I would like this to happen automatically when a job is submitted --
I never used the old GASS cache, but from some Googling perhaps what I'm
proposing is similar. Let's say before I construct my RSL file, I need to
figure out whether or not a file exists on the remote resource. Is there a
way to do this with pure RFT/GridFTP?
Furthermore, is there a way to cause a symlink to be created on a remote
resource using RFT/GridFTP?
I thought I'd start with this list before I submit a message to either the
GRAM or GridFTP lists.
Thanks,
Adam
