Copying data periodically to gateway storage/ user's desktop can be done from MFT if we have following
1. If it's copying to gateway storage, gateway storage side should have a MFT Agent installed 2. If it's for user's desktop, user should have MFT Agent installed and provide write access to a particular directory. 3. However in both cases, we need to have another service instructing to MFT when, what, where to copy at each iteration. 4. In addition to that, we need some changes to Airavata API models to store and configure metadata for periodic synchronizations. I believe this is a good GSoC project if someone is willing to take on and I would like to act as a mentor. Dimuthu On Thu, Mar 26, 2020 at 12:47 PM Pamidighantam, Sudhakar <pamid...@iu.edu> wrote: > I am not suggesting we mount any disk but potentially transfer the data in > that remote HPC disk to storage in the gateway and provide access from > gateway storage or directly to users desktop periodically/or on prompt > during the run. > > > > Thanks, > > Sudhakar. > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Reply-To: *"dev@airavata.apache.org" <dev@airavata.apache.org> > *Date: *Thursday, March 26, 2020 at 12:42 PM > *To: *Airavata Dev <dev@airavata.apache.org> > *Subject: *Re: MFT and data access for running jobs > > > > Sudhakar, > > > > What you are asking is not a direct MFT use case. It's more like a NFS > mount of a remote file system to a local file system. MFT is mainly > focussing on handling the data transfer path not synching data between two > endpoints at realtime. > > > > Thanks > > Dimuthu > > > > On Thu, Mar 26, 2020 at 12:29 PM Pamidighantam, Sudhakar <pamid...@iu.edu> > wrote: > > Dimuthu: > > > > Yes, the working directory on remote HPC cluster. > > > > The workflow may look like this.. > > > > The user launches a job.. > > The remote working directory, dynamically defined by Airavata during the > launch of the experiment is registered as a remote disk accessible > > The contents are made available readonly for users to read/download > > Remove this as accessible when the experiment ends > > Continue with the rest of the Helix tasks > > … > > > > > > Thanks, > > Sudhakar. > > > > *From: *DImuthu Upeksha <dimuthu.upeks...@gmail.com> > *Reply-To: *"dev@airavata.apache.org" <dev@airavata.apache.org> > *Date: *Thursday, March 26, 2020 at 12:23 PM > *To: *Airavata Dev <dev@airavata.apache.org> > *Subject: *Re: MFT and data access for running jobs > > > > Sudhakar, > > I’m not sure whether I grabbed your point about this remote working > directory correctly. Are you taking about the working directory of the > cluster? Can you please explain the workflow with more details? > > Thanks > Dimuthu > > > > On Thu, Mar 26, 2020 at 10:21 AM Pamidighantam, Sudhakar <pamid...@iu.edu> > wrote: > > Dimuthu: > > > > When the MFT becomes available would there be a way to define the remote > working directory as a device to provide access to the data there. > > You know this has been a long standing need for particularly long running > jobs. > > > > Thanks, > > Sudhakar. > > > >
