Re: Python SDK for MFT

[Rishabh Gajra]

Thanks for the suggestions Isuru and Marru.
I got it and I will go ahead and use this method as of now. I will use the
maven plugin that is being used in Airavata-MFT (org.xolstice.maven.plugins)
But I have a question for the future. Like we are building the client with
all the proto classes like in java. We need to find a way to push into the
Pip right? Also in the future we are going to separate out the clients from
the main server repo of Airavata-MFT, aren't we? I will use the method
suggested above as of now. I will later discuss about the Pip later.
Thanks once again for the suggestion!
Rishabh.

On Thu, Apr 30, 2020 at 10:39 PM Suresh Marru  wrote:

> On Apr 30, 2020, at 10:26 PM, Isuru Ranawaka  wrote:
>
>
> Hi Rishabh,
>
> Some suggestions, try out a maven plugin (e.g protbuf-maven-plugin) to
> generate stubs and copy them into relevant folders or write a script (e.g
> shell script) and execute via maven phase.
>
>
> Here is an example shell script we use for generation of Thrift SDK’s. You
> could follow this example and write equivalent protobuff one and link it
> maven phase as Isuru suggested:
>
>
> https://github.com/apache/airavata/blob/master/thrift-interface-descriptions/generate-thrift-stubs.sh
>
> Suresh
>
>
> thanks
> Isuru
>
>
>
>
> On Thu, Apr 30, 2020 at 10:03 PM Rishabh Gajra 
> wrote:
>
> Hi,
> This is related to custos python client that is present in the develop
> branch. I believe all the classes that are present in the module are
> generated from the proto buff. I wanted to address the issue where there is
> a change in the protobuff file on the server-side. Is there any method
> where we can avoid manually putting the class file in every client that we
> build on. Also, our team is planning to write the Python SDK for Airavata
> MFT (Possibly extend to GOLANG too if time permits).
> Specifically, I want to address the above issue. Any suggestion on how to
> go about this is appreciated. Thank you.
>
>
> --
> Regards,
> Rishabh.
>
>
>
> --
> Research Software Engineer
> Indiana University, IN
>
>
>

-- 
Regards,
Rishabh.

Re: [External] Re: Airavata MFT- Gdrive transport implementation

[DImuthu Upeksha]

Hi Shivali,

This looks great. This is a good habit for other transport developers also.
Once we have some quality content, we can directly put into the repository
wiki. I requested some changes to your PR and please update relevant parts
of the document once you addressed them.

Thanks
Dimuthu

On Fri, May 1, 2020 at 1:33 PM Jejurkar, Shivali  wrote:

> Hello,
>
>
>
> We have created a document for setting up service account for Google Drive
> transport setup in MFT. Kindly review  and share your comments/feedback.
>
>
> https://docs.google.com/document/d/1ql3megjxd_4_t9ID23se9Lb8NFpY83e175XHoKYpYdg/edit?usp=sharing
>
>
>
>
>
> Thanks,
>
> Shivali Jejurkar
>
>
>
> Sent from Mail  for
> Windows 10
>
>
>
> *From: *Bafna, Nikita 
> *Sent: *Thursday, April 30, 2020 12:43 PM
> *To: *dev@airavata.apache.org
> *Subject: *RE: [External] Re: Airavata MFT- Gdrive transport
> implementation
>
>
>
> Hi Dimuthu,
>
>
>
> We have now addressed all the issues raised.
>
> Could you please verify and provide your inputs.
>
>
>
> Regards,
>
> Nikita
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Wednesday, April 29, 2020 1:49 AM
> *To: *Airavata Dev 
> *Subject: *Re: [External] Re: Airavata MFT- Gdrive transport
> implementation
>
>
> Still there are some unaddressed reviews available. Please have a look at
> them.
>
> [2] https://github.com/apache/airavata-mft/pull/11/files
>
> On Wed, Apr 29, 2020 at 1:33 AM Patil, Aniruddha Vivek 
> wrote:
>
>> Hello Dimuthu,
>>
>> We addressed the changes suggested in your review. Please let us know
>> what should be addressed further.
>>
>> Thanks and regards,
>> Aniruddha Patil
>>
>> On Apr 28, 2020, at 10:05 PM, DImuthu Upeksha 
>> wrote:
>>
>> Thanks. I did the first pass of you PR. Please have a look. Once you
>> address the issues, I will do more detailed reviews.
>>
>> Dimuthu
>>
>> On Tue, Apr 28, 2020 at 6:22 PM Bafna, Nikita  wrote:
>>
>>> Hello Dimuthu,
>>>
>>>
>>> Sorry I forgot to mention the PR [1] link. Here it is.
>>>
>>>
>>> [1] https://github.com/apache/airavata-mft/pull/11
>>>
>>>
>>> Regards,
>>>
>>> Nikita
>>>
>>>
>>> *From: *Bafna, Nikita 
>>> *Sent: *Tuesday, April 28, 2020 4:39 PM
>>> *To: *dev@airavata.apache.org
>>> *Subject: *Re: [External] Re: Airavata MFT- Gdrive transport
>>> implementation
>>>
>>>
>>> Hello Dimuthu,
>>>
>>> We have implemented transport system for Google drive.
>>>
>>> We have build our transport in a way where if a file with that name
>>> already exists it will update that file instead of creating a new one on
>>> Google drive.
>>>
>>> For now we have tested our implementation between endpoints below.
>>>
>>> S3 to Google drive
>>>
>>> Google drive to S3
>>>
>>> Local to Google drive (Screenshot attached)
>>>
>>> Google drive to Local (Screenshot attached)​
>>>
>>> Could you please have a look and let us know your feedback.
>>>
>>> Thank you,
>>> Team NSA
>>> (Nikita Bafna, Shivali Jejurkar, Aniruddha Patil)
>>>
>>>
>>> --
>>> *From:* Bafna, Nikita
>>> *Sent:* Tuesday, April 28, 2020 5:20 AM
>>> *To:* dev@airavata.apache.org
>>> *Subject:* Re: [External] Re: Airavata MFT- Gdrive transport
>>> implementation
>>>
>>>
>>> Hi Dimuthu,
>>>
>>>
>>> During testing of the GDrive transport, we came across a situation where
>>> if a file is being uploaded on the Gdrive and if the file with same name
>>> already exists, then Gdrive creates a new file with the same name (it
>>> creates a new version ). So how should this be handled? Should it create a
>>> new file or update the existing file?
>>>
>>>
>>> Let us know your inputs.
>>>
>>>
>>> Regards,
>>>
>>> Nikita
>>> --
>>> *From:* Bafna, Nikita 
>>> *Sent:* Tuesday, April 28, 2020 12:48 AM
>>> *To:* dev@airavata.apache.org
>>> *Subject:* RE: [External] Re: Airavata MFT- Gdrive transport
>>> implementation
>>>
>>>
>>> Okay.
>>>
>>> Thanks for the inputs Dimuthu.
>>>
>>>
>>> *From:* DImuthu Upeksha 
>>> *Sent:* Monday, April 27, 2020 3:05 AM
>>> *To:* Airavata Dev 
>>> *Subject:* Re: [External] Re: Airavata MFT- Gdrive transport
>>> implementation
>>>
>>>
>>> Go with service account approach as this is service to service
>>> authentication
>>>
>>>
>>> Dimuthu
>>>
>>>
>>> On Sat, Apr 25, 2020 at 7:11 PM Bafna, Nikita  wrote:
>>>
>>> Hi Dimuthu,
>>>
>>>
>>> While working for Google drive transport, we are wondering what kind of
>>> credential account should we use for our setup.
>>>
>>>
>>> Attaching the options available.
>>>
>>>
>>> Could you please provide your feedback.
>>>
>>>
>>> Regards,
>>>
>>> Nikita
>>> ​
>>> --
>>>
>>> *From:* Suresh Marru 
>>> *Sent:* Friday, April 24, 2020 11:36 PM
>>> *To:* Airavata Dev
>>> *Subject:* [External] Re: Airavata MFT- Gdrive transport implementation
>>>
>>>
>>> This message was sent from a non-IU address. Please exercise caution
>>> when clicking links or opening attachments from external sources.
>>>
>>>
>>> A big kudos for all the 

RE: [External] Re: Airavata MFT- Gdrive transport implementation

[Jejurkar, Shivali]

Hello,

We have created a document for setting up service account for Google Drive 
transport setup in MFT. Kindly review  and share your comments/feedback.
https://docs.google.com/document/d/1ql3megjxd_4_t9ID23se9Lb8NFpY83e175XHoKYpYdg/edit?usp=sharing


Thanks,
Shivali Jejurkar

Sent from Mail for Windows 10

From: Bafna, Nikita
Sent: Thursday, April 30, 2020 12:43 PM
To: dev@airavata.apache.org
Subject: RE: [External] Re: Airavata MFT- Gdrive transport implementation

Hi Dimuthu,

We have now addressed all the issues raised.
Could you please verify and provide your inputs.

Regards,
Nikita

From: DImuthu Upeksha
Sent: Wednesday, April 29, 2020 1:49 AM
To: Airavata Dev
Subject: Re: [External] Re: Airavata MFT- Gdrive transport implementation

Still there are some unaddressed reviews available. Please have a look at them.

[2] https://github.com/apache/airavata-mft/pull/11/files

On Wed, Apr 29, 2020 at 1:33 AM Patil, Aniruddha Vivek 
mailto:anipa...@iu.edu>> wrote:
Hello Dimuthu,

We addressed the changes suggested in your review. Please let us know what 
should be addressed further.

Thanks and regards,
Aniruddha Patil

On Apr 28, 2020, at 10:05 PM, DImuthu Upeksha 
mailto:dimuthu.upeks...@gmail.com>> wrote:

Thanks. I did the first pass of you PR. Please have a look. Once you address 
the issues, I will do more detailed reviews.

Dimuthu

On Tue, Apr 28, 2020 at 6:22 PM Bafna, Nikita 
mailto:niba...@iu.edu>> wrote:
Hello Dimuthu,

Sorry I forgot to mention the PR [1] link. Here it is.

[1] https://github.com/apache/airavata-mft/pull/11

Regards,
Nikita

From: Bafna, Nikita
Sent: Tuesday, April 28, 2020 4:39 PM
To: dev@airavata.apache.org
Subject: Re: [External] Re: Airavata MFT- Gdrive transport implementation


Hello Dimuthu,

We have implemented transport system for Google drive.

We have build our transport in a way where if a file with that name already 
exists it will update that file instead of creating a new one on Google drive.

For now we have tested our implementation between endpoints below.

S3 to Google drive

Google drive to S3

Local to Google drive (Screenshot attached)

Google drive to Local (Screenshot attached)​

Could you please have a look and let us know your feedback.

Thank you,
Team NSA
(Nikita Bafna, Shivali Jejurkar, Aniruddha Patil)



From: Bafna, Nikita
Sent: Tuesday, April 28, 2020 5:20 AM
To: dev@airavata.apache.org
Subject: Re: [External] Re: Airavata MFT- Gdrive transport implementation


Hi Dimuthu,


During testing of the GDrive transport, we came across a situation where if a 
file is being uploaded on the Gdrive and if the file with same name already 
exists, then Gdrive creates a new file with the same name (it creates a new 
version ). So how should this be handled? Should it create a new file or update 
the existing file?


Let us know your inputs.


Regards,

Nikita


From: Bafna, Nikita mailto:niba...@iu.edu>>
Sent: Tuesday, April 28, 2020 12:48 AM
To: dev@airavata.apache.org
Subject: RE: [External] Re: Airavata MFT- Gdrive transport implementation

Okay.
Thanks for the inputs Dimuthu.

From: DImuthu Upeksha 
mailto:dimuthu.upeks...@gmail.com>>
Sent: Monday, April 27, 2020 3:05 AM
To: Airavata Dev mailto:dev@airavata.apache.org>>
Subject: Re: [External] Re: Airavata MFT- Gdrive transport implementation

Go with service account approach as this is service to service authentication

Dimuthu

On Sat, Apr 25, 2020 at 7:11 PM Bafna, Nikita 
mailto:niba...@iu.edu>> wrote:

Hi Dimuthu,



While working for Google drive transport, we are wondering what kind of 
credential account should we use for our setup.



Attaching the options available.



Could you please provide your feedback.



Regards,

Nikita

​

From: Suresh Marru mailto:sma...@apache.org>>
Sent: Friday, April 24, 2020 11:36 PM
To: Airavata Dev
Subject: [External] Re: Airavata MFT- Gdrive transport implementation

This message was sent from a non-IU address. Please exercise caution when 
clicking links or opening attachments from external sources.

A big kudos for all the work on transports, these are nicely evolving.

We do not need to drive these with IU decisions, but good point these will lead 
to potential use cases. On that note, looks like Google Drive will be used for 
non-sensitive data and one drive for PHI data. So an eventual plan for MFT to 
support both will be nice. I added it to the spreadsheet.

Suresh


On Apr 24, 2020, at 12:45 PM, DImuthu Upeksha 
mailto:dimuthu.upeks...@gmail.com>> wrote:

Because of the IUs decision of migration of Box data to Google Drive, I expect 
the use cases will come into MFT to 

[GitHub] [airavata-custos] smarru commented on pull request #10: Custos develop branch updates

[GitBox]

smarru commented on pull request #10:
URL: https://github.com/apache/airavata-custos/pull/10#issuecomment-622287597


   Given the implementations changed significantly, this is no longer relevant.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airavata-custos] smarru commented on pull request #11: User Profile SDK

[GitBox]

smarru commented on pull request #11:
URL: https://github.com/apache/airavata-custos/pull/11#issuecomment-622287523


   Given the implementations changed significantly, this is no longer relevant.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airavata-custos] smarru commented on pull request #12: upgraded java version to 11 from 8

[GitBox]

smarru commented on pull request #12:
URL: https://github.com/apache/airavata-custos/pull/12#issuecomment-622287291


   Given the implementations changed significantly, this is no longer relevant.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airavata-custos] smarru commented on pull request #13: added backend capability to create ci logon client

[GitBox]

smarru commented on pull request #13:
URL: https://github.com/apache/airavata-custos/pull/13#issuecomment-622287128


   Given the implementations changed significantly, this is no longer relevant.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airavata-custos] smarru commented on pull request #15: added APIs for tenant service

[GitBox]

smarru commented on pull request #15:
URL: https://github.com/apache/airavata-custos/pull/15#issuecomment-622287033


   Given the implementations changed significantly, this is no longer relevant.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [External] Apache Airavata MFT - OneDrive transport implementation

[Patil, Aniruddha Vivek]

Hello,

So far, we have implemented a OneDrive transport scaffold for Airavata’s MFT 
[1]. The core logic implementing onedrive-transport depends on being able to 
access OneDrive from Microsoft Graph, the development of which, is on hold. We 
have investigated several ways in order to get this to work.

- Microsoft provides mainly two flows with which an authentication token can be 
generated.
- Regardless of which flow is chosen, all of Microsoft's documentation points 
towards creating an Azure Active Directory (AAD) App on the Azure Portal.
- We have tried creating an AAD app, provided the necessary permissions and 
generated the relevant credentials for it.

The two authentication flows are as follows:
1. OAuth 2.0 authorization code grant flow [2]

We have investigated the following samples in order to be able to generate the 
access token:

1.1. Nuxeo’s onedrive-java-client. [3]
- It does not provide a method to complete OAuth and obtain the access token.

1.2 The example you cited in one of the previous mails. [4]
- It does not provide a method to complete OAuth and obtain the access token.

1.3 The third party client (onedrive-sdk-java) we discussed about. [5]
- Requires spawning an intermediate browser window for authentication.

1.4 Microsoft's own Postman guide [6]
- Generated an authentication token, but we were unable to list our drive files 
due to a license error (attempted license resolution addressed below)

1.5 Azure Free Trial for adding OneDrive to the AAD [7]
- Assuming that the OneDrive service within the AAD was a paid one, we tried 
enabling the provided Azure free subscription.
- We were not able to figure out why an added user within an AAD did not have 
their own dedicated OneDrive.

1.6 To circumvent the intermediate authentication step, we looked at the 
Implicit Grant flow [8] and were unable to obtain an authentication token from 
the /authorize endpoint despite having allowed the same in the AAD application.

1.7 MS Grah Explorer [9] is able to make queries to OneDrive since it is 
already a web application.

2. OAuth 2.0 client credentials flow [10]

2.1 Using this flow grays out the possibility of using the /me alias for 
querying MS Graph and requires the usage of /users/{user_id} instead.

2.2 Thus the enumeration of the files within a user's OneDrive cannot be done 
with /me/drive/root/children like how MS Graph Explorer does it.

We seem to be stuck at emulating queries that MS Graph Explorer does. The 
reason this mail is so long is because wanted to give as much detail as 
possible in order to help arrive at a solution, or atleast provide a list of 
pointers to avoid trying the same methods. We would like to know how we can 
approach this issue.

[1] https://github.com/aniruddhavpatil/airavata-mft/tree/develop
[2] https://docs.microsoft.com/en-us/graph/auth-v2-user
[3] https://github.com/nuxeo/onedrive-java-client
[4] https://www.example-code.com/java/onedrive_download_to_file.asp
[5] https://github.com/isac322/OneDrive-SDK-java
[6] https://docs.microsoft.com/en-us/graph/use-postman
[7] 
https://stackoverflow.com/questions/46802055/tenant-does-not-have-a-spo-license
[8] 
https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-implicit-grant-flow
[9] https://developer.microsoft.com/en-us/graph/graph-explorer
[10] 
https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow

On Apr 28, 2020, at 6:04 PM, Patil, Aniruddha Vivek 
mailto:anipa...@iu.edu>> wrote:

Hello Dimuthu,


  1.  We create an Azure Active Directory application on the Azure Portal [1] 
with credentials for the Airavata organization.
  2.  I have tried out using MS Graph Explorer [2] to access my OneDrive files. 
This functionality will be emulated by our MFT application that will use the 
credentials of the step above.
  3.  Once the validity of the above two steps is confirmed, we can decide on a 
client to use for accessing MS Graph. Microsoft has put up msgraph-sdk-java [3] 
on their website as a Java client as per [4].
  4.  The above steps are my understandings of [5].

[1] https://portal.azure.com/
[2] https://developer.microsoft.com/en-us/graph/graph-explorer
[3] https://github.com/microsoftgraph/msgraph-sdk-java
[4] https://developer.microsoft.com/en-us/graph/get-started/java
[5] https://docs.microsoft.com/en-us/graph/tutorials/java

Regards,
Aniruddha Patil



On Apr 27, 2020, at 11:58 PM, DImuthu Upeksha 
mailto:dimuthu.upeks...@gmail.com>> wrote:

Can you summarize the suggested steps to register an onedrive resource in MFT 
including registering it in ms graph and fetching tokens? I need to have an 
idea on the convenience aspect on this. About the API clients, you should try 
to stick to standard clients provided by original providers as much as 
possible. If not try to go for a reputed client implementation form third 
party. In either case, licenses of these clients should be compatible with 
Apache License 2.0. I just had a