Re: Facility to upload large files to a cluster

[Christie, Marcus Aaron]

Not directly. You would upload the file to data storage then use it as an input 
to an experiment. When the experiment is launched the Airavata middleware will 
stage the file to the cluster as needed.

On Apr 18, 2018, at 3:58 PM, Thejaka Amila J Kanewala 
> wrote:

Hi Marcus,

Thank you for the reply.
I need to get the large file into the cluster. Does uploading file to data 
storage means it will be available in the cluster?
If so, it is sufficient for me.

Regards
-Thejaka

On Mon, Apr 16, 2018 at 11:43 AM, Christie, Marcus Aaron 
> wrote:
Amila,

We don’t really have anything like this right now.  Sachin is working on a new 
approach to data management which might help by allowing users to upload files 
directly to a data store and then selecting those files when configuring an 
experiment. Would that suffice? That’s a little different from your requirement 
to upload a file directly to the cluster.

> On Apr 12, 2018, at 3:17 PM, Thejaka Amila J Kanewala 
> > wrote:
>
> Hello,
>
> I am planning to create a gateway for domain specific usage and I have the 
> following requirement.
>
> Users of this gateway should be able to upload a very large file (probably 
> more than 200GB) to the cluster (probably to the lustre file system within 
> the cluster) and use that file in an experiment.
>
> Ideally, I am looking for some form of a command line tool as follows:
>
> ./airavata-file-upload  
>
> Then this should prompt for
> Gateway user id : kermit
> Gateway Password : ***
> Experiment Id: elmo
> Associated Experiment Attribute: webgraph
>
> Is this possible in current implementation? Sorry, I am not up to date with 
> recent developments in Airavata.
>
> Thanks
> -Thejaka
>
>

Re: Facility to upload large files to a cluster

[Thejaka Amila J Kanewala]

Hi Marcus,

Thank you for the reply.
I need to get the large file into the cluster. Does uploading file to data
storage means it will be available in the cluster?
If so, it is sufficient for me.

Regards
-Thejaka

On Mon, Apr 16, 2018 at 11:43 AM, Christie, Marcus Aaron 
wrote:

> Amila,
>
> We don’t really have anything like this right now.  Sachin is working on a
> new approach to data management which might help by allowing users to
> upload files directly to a data store and then selecting those files when
> configuring an experiment. Would that suffice? That’s a little different
> from your requirement to upload a file directly to the cluster.
>
> > On Apr 12, 2018, at 3:17 PM, Thejaka Amila J Kanewala <
> thejaka.am...@gmail.com> wrote:
> >
> > Hello,
> >
> > I am planning to create a gateway for domain specific usage and I have
> the following requirement.
> >
> > Users of this gateway should be able to upload a very large file
> (probably more than 200GB) to the cluster (probably to the lustre file
> system within the cluster) and use that file in an experiment.
> >
> > Ideally, I am looking for some form of a command line tool as follows:
> >
> > ./airavata-file-upload  
> >
> > Then this should prompt for
> > Gateway user id : kermit
> > Gateway Password : ***
> > Experiment Id: elmo
> > Associated Experiment Attribute: webgraph
> >
> > Is this possible in current implementation? Sorry, I am not up to date
> with recent developments in Airavata.
> >
> > Thanks
> > -Thejaka
> >
> >
>
>

Re: Facility to upload large files to a cluster

[Christie, Marcus Aaron]

Amila,

We don’t really have anything like this right now.  Sachin is working on a new 
approach to data management which might help by allowing users to upload files 
directly to a data store and then selecting those files when configuring an 
experiment. Would that suffice? That’s a little different from your requirement 
to upload a file directly to the cluster.

> On Apr 12, 2018, at 3:17 PM, Thejaka Amila J Kanewala 
>  wrote:
> 
> Hello,
> 
> I am planning to create a gateway for domain specific usage and I have the 
> following requirement.
> 
> Users of this gateway should be able to upload a very large file (probably 
> more than 200GB) to the cluster (probably to the lustre file system within 
> the cluster) and use that file in an experiment.
> 
> Ideally, I am looking for some form of a command line tool as follows:
> 
> ./airavata-file-upload  
> 
> Then this should prompt for 
> Gateway user id : kermit
> Gateway Password : ***
> Experiment Id: elmo
> Associated Experiment Attribute: webgraph 
> 
> Is this possible in current implementation? Sorry, I am not up to date with 
> recent developments in Airavata.
> 
> Thanks
> -Thejaka
> 
>