Re: [em-center-gateway] MFT requirements for EMC portal

[DImuthu Upeksha]

Hi Dinuka,

Sorry for the late reply. Please find my reply inline. Hope that helps.
However it would be great if you can come up with a simple component /
architecture diagram that visualizes how these components might connect to
each other. It doesn't have to be perfect but atleast we might be able to
help you to figure out missing pieces.

Thanks
Dimuthu

On Tue, Oct 20, 2020 at 9:01 AM Dinuka Desilva 
wrote:

>
>
> On Mon, Oct 19, 2020 at 9:07 PM Dinuka Desilva 
> wrote:
>
>> Hi Dimuthu,
>>
>> Let me bring back the questions we discussed at the call to a discussion
>> thread. We currently see the following requirements from MFT. Let's discuss
>> further here.
>>
>> 1) Transferring files from the microscopes to the users storage. In this,
>> there's no authentication layer between the users other than a root level
>> authentication (if there's anything possible). The user and folder
>> structure has to be understood from the folder or file path specified at
>> the microscope.
>>
>> Eg: if the file path at the microscope is
>> "./abce-microscope/dinuka/session-36/bacteria-sample.png", It should be
>> uploaded to the user "dinuka" to the folder "./abce-microscope/session-36"
>> in his user drive.
>>
>> There should be another intelligent layer that should instruct MFT to do
this. You have to specify 2 resources for source and destination in MFT and
then invoke the transfer API to transfer those files.

@Marru, Suresh  @Kuruvilla, Tanya Elizabeth  I
>> would +1 for above. I believe it's good to let the users share the files to
>> the groups manually instead of taking that also from the microscope file
>> path. In my perspective, I would like not to share directly. Instead I
>> would like to go back to my computer and see the files and do any
>> modifications if needed and then share with the PI groups.
>>
>> 2) For file management, there's the need of managing folder structure for
>> individual users. And the users should be able to share the files or
>> folders to individual users or groups.
>>
>> @Christie, Marcus Aaron  I think for the user storage
>> management within the diravata-django-portal also, we could use MFT if
>> these requirements are satisfied from MFT.
>>
>
> Looking at the intention of MFT, I wonder if this is outside MFT scope. I
> believe there's no permanent storage of the files in MFT. Instead, MFT is a
> managed bridge for transferring files between two locations that could be
> from the same or different protocols.
>
>
You are correct. MFT is stateless and specialized for transferring files.
This part goes to the sharing aspect of Custos.

@isjar...@iu.edu  @Marru, Suresh 
> I was just thinking whether it's through Custos the user file storage has
> to be there. Or even as in like a seperate service providing storage for
> every Custos user.
>
>
>> 3) Form the microscopes, since there's no way of getting individual user
>> authentication processed and it has to be deeply coupled with the path of
>> the files created (1) and also since the creation of the files is
>> continuous ,there's a need of having a folder syncing capability which has
>> to keep watching the file changes.
>>
>> Regards,
>> Dinuka
>>
>>

Re: [em-center-gateway] MFT requirements for EMC portal

[Christie, Marcus Aaron]

> On Oct 20, 2020, at 9:00 AM, Dinuka Desilva  wrote:
> 
> Looking at the intention of MFT, I wonder if this is outside MFT scope. I 
> believe there's no permanent storage of the files in MFT. Instead, MFT is a 
> managed bridge for transferring files between two locations that could be 
> from the same or different protocols. 
> 

Hi Dinuka,

I'm wondering the same thing. If it is in scope for MFT to list directories and 
provide directory management in addition to getting and putting files, then we 
could use the user storage management UIs in the Django portal. Those UIs 
leverage the user_storage SDK module on the backend [1] so if the user_storage 
functions can be implemented using the MFT API it would work.

[1] 
https://airavata-django-portal-sdk.readthedocs.io/en/latest/#module-user_storage
 




smime.p7s
Description: S/MIME cryptographic signature

Re: [em-center-gateway] MFT requirements for EMC portal

[Dinuka Desilva]

On Mon, Oct 19, 2020 at 9:07 PM Dinuka Desilva 
wrote:

> Hi Dimuthu,
>
> Let me bring back the questions we discussed at the call to a discussion
> thread. We currently see the following requirements from MFT. Let's discuss
> further here.
>
> 1) Transferring files from the microscopes to the users storage. In this,
> there's no authentication layer between the users other than a root level
> authentication (if there's anything possible). The user and folder
> structure has to be understood from the folder or file path specified at
> the microscope.
>
> Eg: if the file path at the microscope is
> "./abce-microscope/dinuka/session-36/bacteria-sample.png", It should be
> uploaded to the user "dinuka" to the folder "./abce-microscope/session-36"
> in his user drive.
>
> @Marru, Suresh  @Kuruvilla, Tanya Elizabeth
>  I would +1 for above. I believe it's good to let the
> users share the files to the groups manually instead of taking that also
> from the microscope file path. In my perspective, I would like not to share
> directly. Instead I would like to go back to my computer and see the files
> and do any modifications if needed and then share with the PI groups.
>
> 2) For file management, there's the need of managing folder structure for
> individual users. And the users should be able to share the files or
> folders to individual users or groups.
>
> @Christie, Marcus Aaron  I think for the user storage
> management within the diravata-django-portal also, we could use MFT if
> these requirements are satisfied from MFT.
>

Looking at the intention of MFT, I wonder if this is outside MFT scope. I
believe there's no permanent storage of the files in MFT. Instead, MFT is a
managed bridge for transferring files between two locations that could be
from the same or different protocols.

@isjar...@iu.edu  @Marru, Suresh 
I was just thinking whether it's through Custos the user file storage has
to be there. Or even as in like a seperate service providing storage for
every Custos user.


> 3) Form the microscopes, since there's no way of getting individual user
> authentication processed and it has to be deeply coupled with the path of
> the files created (1) and also since the creation of the files is
> continuous ,there's a need of having a folder syncing capability which has
> to keep watching the file changes.
>
> Regards,
> Dinuka
>
>

[em-center-gateway] MFT requirements for EMC portal

[Dinuka Desilva]

Hi Dimuthu,

Let me bring back the questions we discussed at the call to a discussion
thread. We currently see the following requirements from MFT. Let's discuss
further here.

1) Transferring files from the microscopes to the users storage. In this,
there's no authentication layer between the users other than a root level
authentication (if there's anything possible). The user and folder
structure has to be understood from the folder or file path specified at
the microscope.

Eg: if the file path at the microscope is
"./abce-microscope/dinuka/session-36/bacteria-sample.png", It should be
uploaded to the user "dinuka" to the folder "./abce-microscope/session-36"
in his user drive.

@Marru, Suresh  @Kuruvilla, Tanya Elizabeth  I
would +1 for above. I believe it's good to let the users share the files to
the groups manually instead of taking that also from the microscope file
path. In my perspective, I would like not to share directly. Instead I
would like to go back to my computer and see the files and do any
modifications if needed and then share with the PI groups.

2) For file management, there's the need of managing folder structure for
individual users. And the users should be able to share the files or
folders to individual users or groups.

@Christie, Marcus Aaron  I think for the user storage
management within the diravata-django-portal also, we could use MFT if
these requirements are satisfied from MFT.

3) Form the microscopes, since there's no way of getting individual user
authentication processed and it has to be deeply coupled with the path of
the files created (1) and also since the creation of the files is
continuous ,there's a need of having a folder syncing capability which has
to keep watching the file changes.

Regards,
Dinuka