Re: [ANNOUNCE] Welcome Lahiru Jayathilake as Airavata PMC member/committer

[Dimuthu Upeksha]

Welcome, Lahiru. Looking forward for your continuing contributions

Thanks
Dimuthu

On Mon, Apr 8, 2024 at 5:43 PM Suresh Marru  wrote:

> Hi All,
>
> The Project Management Committee (PMC) for Apache Airavata has asked Lahiru 
> Jayathilake to become a committer and PMC member based on his contributions 
> to the project. We are pleased to announce that he has accepted.
>
> Being a committer enables easier contribution to the project since there is 
> no need to go via the patch submission process. This should enable better 
> productivity. Being a PMC member enables assistance with the management and 
> to guide the direction of the project.
>
> Please join me in welcoming Lahiru to Airavata.
>
> Suresh
> (On Behalf of Apache Airavata PMC)
>
>

Re: Airavata-MFT | Local transport implementation

[Dimuthu Upeksha]

Hi Praneeth,

SCP transport is now available in MFT [1] and working fine with S3. You can
update your load testing scripts to measure the performance.

[1]
https://github.com/apache/airavata-mft/commit/a61fa4a34129cf0e7e807ee23461237a8a72af22

Thanks
Dimuthu

On Mon, Mar 20, 2023 at 3:43 PM Chityala, Praneeth  wrote:

> Hi Marcus,
>
> Thank you. The Local plugin to download data has been implemented, but the
> testing to check speeds is something not done yet. I will also add this to
> my future performance testing.
>
> Best,
> Praneeth
>
> On 3/20/23, 2:43 PM, "Christie, Marcus Aaron"  machr...@iu.edu>> wrote:
>
>
> HI Praneeth,
>
>
> This looks like a great contribution to MFT and I appreciate your write up
> here.
>
>
> One question: these numbers are for uploading from the local EC2 instance
> to S3, correct? Did you do any analysis on the opposite, downloading from
> S3 to a local EC2 instance?
>
>
> Thanks,
>
>
> Marcus
>
>
> > On Mar 17, 2023, at 12:22 AM, Chityala, Praneeth  > wrote:
> >
> > You don't often get email from pkchi...@iu.edu .
> Learn why this is important
> > Dear All,
> >
> > This is Praneeth Chityala currently pursuing master in Computer Science
> at Indiana University Bloomington. As part of my independent study I took
> up MFT as the research area and starting understanding the architecture.
> >
> > As many of you know MFT uses Agent to transfer data from one cloud
> storage to other cloud storage. These agents can be deployed on any compute
> machines. If the machine in which agent is deployed might have data files
> which needs to be uploaded to cloud storage, that’s where my involvement in
> the project came in. I worked on implementing the below extensions:
> > • Implemented the Local transport extension to allow agent to transfer
> data from its host machine given storage – Local transport extension
> > • Transport has three variations – streaming, chunked file transfer and
> chunked streaming
> > • Implemented the CLI for configuring local agent – Local agent CLI
> >
> > Performance testing results:
> >
> > After successfully testing from my local machine to AWS S3 storage, I
> have deployed agent in AWS EC2 machine and performed multiple tests for
> compare it’s performance with rclone and AWS cli.
> > Below charts indicates the average transfer speeds from our analysis.
> >
> > 
> >
> >
> > For files from 100MB to 1GB, MFT is more than 60% faster than rclone and
> more than 150% faster than AWS cli.
> >
> > Configurations of the testing:
> >
> > • Local Machine: It’s Ubuntu EC2 VM on AWS (instance type c5.9xlarge)
> with 18 cores, 10Gbps dedicated network speed and 1GBps read/write speed to
> disk.
> >
> > • Cloud Storage: AWS S3 bucket in the same region as above VM.
> >
> > • Test sets: From x-axis labels of the graph, 10m_1000 means a test set
> of 1000 10MB files. All other test sets follow similar naming convention.
> >
> > • Testing trails: Each test is run for 5 times on each transfer method.
> >
> > • Testing presets: Before each test caching of VM is cleared so none of
> the tests get advantage of higher read speeds using page caching. This is
> done to simulate worst possible conditions while reading data.
> >
> > • MFT configuration: I used chunked streaming with
> > • 20MB as chunk size
> > • 32 concurrent transfers
> > • 32 concurrent chunked threads
> >
> > • rclone configuration: After exploring many possible optimizations
> available for rclone I used following settings:
> > • --s3-chunk-size 128000
> > • --buffer-size 128000
> > • —s3-upload-cutoff 0
> > • --s3-upload-concurrency 32
> > • --multi-thread-streams 32
> > • --multi-thread-cutoff 0
> > • --s3-disable-http2
> > • --no-check-dest
> > • --transfers 32
> > • --fast-list
> >
> > • AWS cli configuration: I used native AWS cli to transfer as it doesn’t
> have much dedicated optimizations in our findings
> >
> > Observations:
> > • For local transport I used BufferedStreaming which helped MFT to get
> the max read speeds from local disk without hitting the max IOPS.
> >
> > Future plans for testing:
> > • Jetstream2: Planning to replace AWS EC2 with Jetstream2 virtual
> machine and perform similar tests
> > • Emulab: Simulate same testing using Emulab VMs and custom
> configurations with help of Dimuthu.
> > • Azure: Perform local to Azure cloud storages testing with MFT, rclone
> and Azure cli
> > • GCP: Perform local to GCS testing with MFT, rclone and GCP cli
> > • I have different implementation of MFT local transport for system
> which support DMA (Direct Memory Access), we also plan to test on such
> systems with DMA, the present EC2 system doesn’t support DMA.
> >
> > Further Improvements of MFT:
> > • As we noticed MFT is lagging speeds vs rclone for files less than or
> equal to 1MB, we plan to stress analyze the whole system and improve speeds
> for smaller files
> >
> > Acknowledgement: I thank Dimuthu Wannipurage for clearing many doubts
> 

Re: Cybershuttle Replica Catalog API

[DImuthu Upeksha]

Hi Jayan,

This looks great and the APIs are clear enough to understand. Specifically,
automatic documentation generation is a very nice feature. I would like to
suggest a few modifications to the replica data models though.

I understand that you got an idea about the replica data models through
Airavata but there are a few modifications/generalizations that we need to
do here. For example, the Replica location category [6] might not make
sense here as you already specify the storage type. UserInfo, GroupInfo,
and Permissions in the same proto file do not apply to this design as those
are handled at the Data Catalog level. What is your expectation in
ReplicaListEntry below?

message ReplicaListEntry {
  string data_replica_id = 1;
  string replica_name = 2;
  StorageType storage_type = 3;
}

If you are planning to provide the grouping of Replica Items through that,
I suggest updating it in the following way

message ReplicaGroupEntry {
  string replica_group_id = 1
  repeated ReplicaGroupEntry directories = 2;
  repeated DataReplicaLocation files = 3;
}

This will provide both grouping and hierarchical replica registration and
you can emulate it as a virtual file hierarchy.

[6]
https://github.com/Jayancv/airavata-replica-catalog/blob/master/replica-catalog-api/stubs/src/main/proto/catalogapi/ReplicaCatalogAPI.proto#L60

Thanks
Dimuthu


On Sat, Feb 18, 2023 at 3:10 PM Jayan Vidanapathirana <
jcvidanapathir...@gmail.com> wrote:

> Hi All,
>
> As a new contributor to the Cybershuttle project, I have been actively
> involved in implementing the Data Replica Catalog. This new catalog is
> designed to interface with both the Apache Airavata Data Catalog [1] and
> Airavata MFT[2]. This replica catalog should be able to store each replica
> resource storage details and secret/credential details specific to the
> storage type. The proposed high-level architecture will be as follows:
>
>
> I will mainly work on the highlighted area (red color box) and as an
> initial step started defining APIs which communicate with Replica Catalog.
> This API calls will be gRPC APIs and following methods will be implement,
>
> Replica Registration
>
>
>1.
>
>registerReplicaLocation(DataReplicaCreateRequest createRequest)
>2.
>
>updateReplicaLocation(DataReplicaCreateRequest updateRequest)
>3.
>
>DataReplicaLocationModel getReplicaLocation(DataReplicaGetRequest
>getReplicaRequest)
>4.
>
>removeReplicaLocation(DataReplicaDeleteRequest deleteReplicaRequest)
>5.
>
>getAllReplicaLocations(AllDataReplicaGetRequest allDataGetRequest)
>6.
>
>removeAllReplicaLocations(AllDataReplicaDeleteRequest
>allDataDeleteRequest)
>
>
> Storage Registration
>
> registerSecretForStorage(SecretForStorage request)
>
> deleteSecretsForStorage(SecretForStorageDeleteRequest request)
>
> getSecretForStorage(SecretForStorageGetRequest request)
>
> searchStorages(StorageSearchRequest request)
>
> listStorages(StorageListRequest request)
>
> resolveStorageType (StorageTypeResolveRequest request)
>
> Storage - Internal APIs
>
> S3StorageListResponse listS3Storage(S3StorageListRequest request)
>
> Optional getS3Storage(S3StorageGetRequest request)
>
> S3Storage createS3Storage(S3StorageCreateRequest request)
>
> boolean updateS3Storage(S3StorageUpdateRequest request)
>
> boolean deleteS3Storage(S3StorageDeleteRequest request)
>
> AzureStorageListResponse listAzureStorage(AzureStorageListRequest request)
>
> Optional getAzureStorage(AzureStorageGetRequest request)
>
> AzureStorage createAzureStorage(AzureStorageCreateRequest request)
>
> boolean updateAzureStorage(AzureStorageUpdateRequest request)
>
> boolean deleteAzureStorage(AzureStorageDeleteRequest request)
>
> GCSStorageListResponse listGCSStorage(GCSStorageListRequest request)
>
> Optional getGCSStorage(GCSStorageGetRequest request)
>
> GCSStorage createGCSStorage(GCSStorageCreateRequest request)
>
> boolean updateGCSStorage(GCSStorageUpdateRequest request)
>
> boolean deleteGCSStorage(GCSStorageDeleteRequest request)
>
> Secret Registration
>
> registerSecret(SecretRegistrationRequest request)
>
> deleteSecret(SecretDeleteRequest request)
>
> resolveStorageType (StorageTypeResolveRequest request)
>
> Secret - Internal APIs
>
>
> Optional getS3Secret(S3SecretGetRequest request)
>
> S3Secret createS3Secret(S3SecretCreateRequest request)
>
> boolean updateS3Secret(S3SecretUpdateRequest request)
>
> boolean deleteS3Secret(S3SecretDeleteRequest request)
>
> Optional getAzureSecret(AzureSecretGetRequest request)
>
> AzureSecret createAzureSecret(AzureSecretCreateRequest request)
>
> boolean updateAzureSecret(AzureSecretUpdateRequest request)
>
> boolean deleteAzureSecret(AzureSecretDeleteRequest request)
>
> Optional getGCSSecret(GCSSecretGetRequest request)
>
> GCSSecret createGCSSecret(GCSSecretCreateRequest request)
>
> boolean updateGCSSecret(GCSSecretUpdateRequest request)
>
> boolean deleteGCSSecret(GCSSecretDeleteRequest request)
>
>
> Poc[3] : 

Re: Regarding Pre and Post job commands in Airavata

[DImuthu Upeksha]

Hi Dinuka,

Sorry for the late reply. It is great to explore options to integrate Dag
capabilities into the job submitter. We already have some form of HTCondor
support in Airavata. Can you summarize the difference between what we
already have for HTCondor and this Dag feature? I am specifically looking
for practical usages instead of technical differences.

Thanks
Dimuthu

On Sun, Jan 1, 2023 at 10:35 AM Dinuka De Silva 
wrote:

> Hi,
>
> The current implementation of mapping these (Pre, Post, etc. job commands)
> to the job scheduler script has assumed the job scheduler script to be a
> type of shell script. So, the order of the execution is based on the order
> of the commands listed in the script which is as below.
>
> - Module Commands
> - Pre Job Commands
> - Job Submitter Command
> - Post Job Commands
>
> The scheduler script of SLURM, FORK, LSF, UGE, and PBS are shell scripts
> while HTCondor and maybe some other job schedulers have different file
> types. The script grammar in HTCondor does not support appending
> shell scripts inside. Now, we are needing to support Pre, Post, and other
> commands for HTCondor realizing the current design doesn't support it.
>
> In HTCondor there's an option [1] to configure pre and post-scripts to be
> executed at the worker instance. But then the script has to be Dag and the
> pre-script, post-script and job-script are to be separate files. So, I
> tried a sample and planning to put this to airavata.
>
> [1]
> https://htcondor.readthedocs.io/en/latest/users-manual/dagman-workflows.html
>
> Thanks & Regards,
> Dinuka
>
>
>

Re: Custos baremetal deployment - need domain names for jetstream2 VMs

[DImuthu Upeksha]

Hi Abhinav,

If you have created VMs through Exosphere, there is an inbuilt domain name
mapping mechanism called UAP[1]. You can directly use that and it comes
with certificates configured. Let me know if you do not understand the
naming convention


[1]
https://gitlab.com/exosphere/exosphere/-/blob/master/docs/user-app-proxy.md

Thanks
Dimuthu

On Wed, Jul 20, 2022 at 9:33 PM Abhinav Sinha 
wrote:

> Hi Eroma,
>
>
>
> I am looking to generate SSL certificates for three vms needed to test
> Custos bare-metal deployment. To do this, I need to register domain names
> for these three jetstream2 ubuntu vms:
>
>
>
> 149.165.152.226
>
> 149.165.154.45
>
> 149.165.152.76
>
>
>
> Could you pls help me with this?
>
>
>
> Thanks,
>
> Abhinav
>

Re: DockerSpawnner for Jupyterhub

[DImuthu Upeksha]

Hi Himanshu,

Use the standard batch spawner provided by jupyterehub. Once you have it
running in a SLURM cluster, you need to change the slurm script that spins
up notebook processes to spinup notebook containers instead. The container
image should be configurable.

[1] https://github.com/jupyterhub/batchspawner

Thanks
Dimuthu

On Fri, Jul 1, 2022 at 11:43 PM Hansaria, Himanshu  wrote:

> I plan to setup slurm spawner following this.
>
> Please let me know if i should be following this for the same.
>
> https://github.com/mkgilbert/slurmspawner
> 
> GitHub - mkgilbert/slurmspawner: Custom Spawner for Jupyterhub to start
> slurm jobs when users log in 
> slurmspawner for Jupyterhub. This is a custom spawner for Jupyterhub that
> is designed for installations on clusters using Slurm scheduling software.
> github.com
> **
> **
> **
> For others, the idea is to instantiate single user notebook servers inside
> a container on the local machine with the Docker Spawner. The eventual
> goal is to write a custom spawner which is a merge of the Docker and the
> slurm Spawner.
>
>
> Thanks and regards,
> Himanshu
> --
> *From:* Hansaria, Himanshu 
> *Sent:* Friday, July 1, 2022 10:54 PM
> *To:* dev@airavata.apache.org 
> *Subject:* [External] Re: DockerSpawnner for Jupyterhub
>
> This message was sent from a non-IU address. Please exercise caution when
> clicking links or opening attachments from external sources.
>
> PS :- I'm able to run a notebook inside it as well.
>
> Thanks,
> Himanshu
> --
> *From:* Hansaria, Himanshu 
> *Sent:* Friday, July 1, 2022 10:48 PM
> *To:* dev@airavata.apache.org 
> *Subject:* [External] DockerSpawnner for Jupyterhub
>
> This message was sent from a non-IU address. Please exercise caution when
> clicking links or opening attachments from external sources.
>
> Hi Dimuthu,
>
> I was able to get the DockerSpawner for jupyterhub working.
>
> Can you confirm if this is the needed functionality?
> Please see the screenshot attached for reference.
>
> My container spawned using DockerSpawner is running on
> http://localhost:53320/.
>
> Thanks and regards,
> Himanshu
>
>
>
>

Re: Regarding seagrid desktop client molecule viewer inetgration

[DImuthu Upeksha]

Hi Aishwarya,

This is a very solid improvement. Is there a way we can try these out? Do
you have a working fork?

Thanks
Dimuthu

On Fri, Jul 1, 2022 at 12:59 PM Pamidighantam, Sudhakar 
wrote:

> Aishwarya:
>
>
>
> Looks good. Keep this and then look at JS molecular Editors or third party
> editors such as Avagadro, or Ovito. Let us plan to meet next week to look
> at these and discuss further.
>
>
>
> Thanks,
>
> Sudhakar.
>
>
>
> *From: *Aishwarya Sinhasane 
> *Reply-To: *"dev@airavata.apache.org" 
> *Date: *Friday, July 1, 2022 at 12:02 PM
> *To: *"dev@airavata.apache.org" 
> *Subject: *Regarding seagrid desktop client molecule viewer inetgration
>
>
>
> Hello Sudhakar
>
>
>
> I was working on integrating a molecule viewer in seagrid desktop client.
> I used the NGL Viewer. I tried various other molecule viewers developed in
> javascript but some of them are not working properly so I choose NGLViewer.
> This molecule viewer supports mmCIF, PDB, mol2, SDF, MMTF structures.
>
>
>
> I am trying to implement a JSMol molecule viewer. Please refer to the
> attached screenshots.
>

Re: Airavata Django Portal

[DImuthu Upeksha]

Hi Lahiru,

Good investigation. If you can make it work with two separate key stores,
please go ahead and do that one until we bump up the keycloak version

Thanks
Dimuthu

On Thu, Jun 30, 2022 at 2:56 PM Lahiru Jayathilake <
lahirujayathil...@gmail.com> wrote:

> Hi Dimuthu,
>
> Thanks for the help via the call to identify the issue was related to the
> keystores. I did a bit of a search and found that the default encryption
> algorithm used in a PKCS12 keystore has been updated on JDK 11.0.12 (and
> up).[1],[2],[3],[4] Since keycloak:2.5.4.Final is using JDK 1.8.0_111, it
> cannot handle a keystore that is generated by 11.0.12+.
>
> [1] -
> https://www.oracle.com/java/technologies/javase/11-0-12-relnotes.html#JDK-8153005
> [2] - https://bugs.openjdk.org/browse/JDK-8202837
> [3] - https://bugs.openjdk.org/browse/JDK-8264362
> [4] - https://bugs.openjdk.org/browse/JDK-8242069
>
> Thanks
>
> On Mon, Jun 27, 2022 at 11:58 PM Lahiru Jayathilake <
> lahirujayathil...@gmail.com> wrote:
>
>> Hi Dimuthu,
>>
>> Performed a maven build and it was successful. And I tried it again.
>> However, I'm getting the same error.
>> Herewith I've attached the logs
>>
>> On Mon, Jun 27, 2022 at 11:16 PM DImuthu Upeksha <
>> dimuthu.upeks...@gmail.com> wrote:
>>
>>> Seems like your API server is not running properly. There are
>>> unoptimized JPA classes. Can you do a maven build again and run
>>> ApiServerStarter again?
>>>
>>> Thanks
>>> Dimuthu
>>>
>>> On Mon, Jun 27, 2022 at 1:40 PM Lahiru Jayathilake <
>>> lahirujayathil...@gmail.com> wrote:
>>>
>>>> Hi Dimuthu,
>>>>
>>>> When I telnet to airavata.host:8443 I get the following. And I've
>>>> attached the logs as well. Hope these help. Thanks in advance.
>>>>
>>>> [image: Screenshot 2022-06-27 at 22.47.59.png]
>>>>
>>>> On Sun, Jun 26, 2022 at 9:52 PM DImuthu Upeksha <
>>>> dimuthu.upeks...@gmail.com> wrote:
>>>>
>>>>> Hi Lahiru,
>>>>>
>>>>> Thanks for the PR. I just merged it. For the error in django portal,
>>>>> you need to verify whether you can telnet to airavata.host:8443. If not,
>>>>> there is some issue in API Server. Please send API server logs as well
>>>>>
>>>>> Dimuthu
>>>>>
>>>>> On Sun, Jun 26, 2022 at 7:06 AM Lahiru Jayathilake <
>>>>> lahirujayathil...@gmail.com> wrote:
>>>>>
>>>>>> Hello Everyone,
>>>>>>
>>>>>> A PR was created addressing the following issues. (
>>>>>> https://github.com/apache/airavata/pull/306)
>>>>>> * Due to some recent security changes were done to "less secure Gmail
>>>>>> accounts" on 30-May-2022[1] (Updated the
>>>>>> airavata/modules/ide-integration/README.md)
>>>>>> * An Enum error when running the JobMonitorStarter (updated the
>>>>>> email-config.yaml file)
>>>>>>
>>>>>> I was following
>>>>>> https://github.com/apache/airavata/tree/develop/modules/ide-integration 
>>>>>> doc
>>>>>> and when I tried to log in to the portal using "default-admin/123456"
>>>>>> credentials I'm getting the following error.
>>>>>>
>>>>>> HTTPSConnectionPool(host='airavata.host', port=8443): Max retries
>>>>>>> exceeded with url: /auth/realms/default/protocol/openid-connect/token
>>>>>>> (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of 
>>>>>>> protocol
>>>>>>> (_ssl.c:1122)')))
>>>>>>
>>>>>>
>>>>>> Full log - https://dpaste.com/3MLV5CSKH
>>>>>>
>>>>>> Does anyone know how to resolve this?
>>>>>>
>>>>>> [1] - https://support.google.com/accounts/answer/6010255?hl=en
>>>>>>
>>>>>> Thanks,
>>>>>> Lahiru
>>>>>>
>>>>>>
>>>>>> On Fri, Jun 24, 2022 at 2:24 AM Lahiru Jayathilake <
>>>>>> lahirujayathil...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Everyone,
>>>>>>>
>>>>>>> I've been going through the following documentation to set up a
>>>>>>> local Airavata Django develo

Re: Abhinav's GSoC progress June 27

[DImuthu Upeksha]

Hi Abhinav,

Thanks for the update. It would be great if there is a working PR for
ansible changes

Thanks
Dimuthu

On Mon, Jun 27, 2022 at 10:06 PM Abhinav Sinha 
wrote:

> Hi Dev,
>
>
>
> Over the last couple of weeks – I was shadowing Isuru’s work on Custos
> bare metal deployment – especially the effort to merge integration
> microservices into one jar.
>
>
>
> Currently, I am working on writing Ansible scripts for deploying these
> changes. I am using the scripts from Airavata-data-lake as reference. Once
> deployed successfully, I plan to do a profiling of the application.
>
>
>
> Thanks,
>
> Abhinav
>

Re: [jupyter-platform-dashboard] No database migration script available

[DImuthu Upeksha]

Hi Dinuka,

Please use the new code base at [3]. Start the API by running it inside the
IDE. Right-click on Application.java and run should work. UI can be started
by running npm start.

[3]
https://github.com/SciGaP/iPython-Kerner-Changes/tree/main/jupyter-platform

Thanks
Dimuthu

On Mon, Jun 27, 2022 at 8:54 PM Dinuka De Silva 
wrote:

> Hi Himanshu,
>
> I tried setting up your project. I don't find a DB script or a migration
> script to create the database. Could you provide me either please? so it's
> helpful for me to provide you with any guidance. I would recommend adding a
> docker-compose to spin up a MySQL docker container, and that way you could
> even keep the database scripts and data also in the repository for anyone
> getting started. Here's [2] a reference you could follow.
>
> Also, I'm using a MacBook and I had been getting the following issue in
> the backend.
>
> ModuleNotFoundError: No module named 'MySQLdb' mac
>
> I was able to fix it by doing the following. [1] while I'm not sure if
> "PyMySQL" is packed with "SQLAlchemy" or if this is a mac only issue.
>
> pip install PyMySQL
>
> // Added these two lines to app.py
> import pymysql
> pymysql.install_as_MySQLdb()
>
> Thanks & Regards,
> Dinuka
>
> [1]
> https://stackoverflow.com/questions/39574813/error-loading-mysqldb-module-no-module-named-mysqldb
> [2]
> https://github.com/apache/airavata/blob/master/modules/ide-integration/src/main/containers/docker-compose.yml#L17
>

Re: Airavata Django Portal

[DImuthu Upeksha]

Seems like your API server is not running properly. There are unoptimized
JPA classes. Can you do a maven build again and run ApiServerStarter again?

Thanks
Dimuthu

On Mon, Jun 27, 2022 at 1:40 PM Lahiru Jayathilake <
lahirujayathil...@gmail.com> wrote:

> Hi Dimuthu,
>
> When I telnet to airavata.host:8443 I get the following. And I've attached
> the logs as well. Hope these help. Thanks in advance.
>
> [image: Screenshot 2022-06-27 at 22.47.59.png]
>
> On Sun, Jun 26, 2022 at 9:52 PM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com> wrote:
>
>> Hi Lahiru,
>>
>> Thanks for the PR. I just merged it. For the error in django portal, you
>> need to verify whether you can telnet to airavata.host:8443. If not, there
>> is some issue in API Server. Please send API server logs as well
>>
>> Dimuthu
>>
>> On Sun, Jun 26, 2022 at 7:06 AM Lahiru Jayathilake <
>> lahirujayathil...@gmail.com> wrote:
>>
>>> Hello Everyone,
>>>
>>> A PR was created addressing the following issues. (
>>> https://github.com/apache/airavata/pull/306)
>>> * Due to some recent security changes were done to "less secure Gmail
>>> accounts" on 30-May-2022[1] (Updated the
>>> airavata/modules/ide-integration/README.md)
>>> * An Enum error when running the JobMonitorStarter (updated the
>>> email-config.yaml file)
>>>
>>> I was following
>>> https://github.com/apache/airavata/tree/develop/modules/ide-integration doc
>>> and when I tried to log in to the portal using "default-admin/123456"
>>> credentials I'm getting the following error.
>>>
>>> HTTPSConnectionPool(host='airavata.host', port=8443): Max retries
>>>> exceeded with url: /auth/realms/default/protocol/openid-connect/token
>>>> (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol
>>>> (_ssl.c:1122)')))
>>>
>>>
>>> Full log - https://dpaste.com/3MLV5CSKH
>>>
>>> Does anyone know how to resolve this?
>>>
>>> [1] - https://support.google.com/accounts/answer/6010255?hl=en
>>>
>>> Thanks,
>>> Lahiru
>>>
>>>
>>> On Fri, Jun 24, 2022 at 2:24 AM Lahiru Jayathilake <
>>> lahirujayathil...@gmail.com> wrote:
>>>
>>>> Hi Everyone,
>>>>
>>>> I've been going through the following documentation to set up a local
>>>> Airavata Django development environment and try out the given samples.
>>>>
>>>> *
>>>> https://github.com/apache/airavata/tree/develop/modules/ide-integration
>>>> * https://github.com/apache/airavata-django-portal#getting-started
>>>> *
>>>> https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/new_django_app/
>>>>
>>>> Due to some code changes, one of the docs needed to be updated.
>>>> I've already created a pull request for that -
>>>> https://github.com/apache/airavata-django-portal/pull/105
>>>>
>>>>
>>>> When starting the API server (according to this
>>>> <https://github.com/apache/airavata/tree/develop/modules/ide-integration>
>>>> guide) I encountered an issue with the IntelliJ Idea version 2022.1.3.
>>>> (MacOS)
>>>>
>>>> Exception - Module 'platform-monitoring' production:
>>>>> java.lang.NoClassDefFoundError: com/intellij/openapi/vfs/VfsUtil
>>>>
>>>>
>>>> Tried updating the relevant plugins and a few ways suggested in the
>>>> Jetbrains issue track. However, the only way I could resolve this was to
>>>> downgrade the Intellij Idea. (working for 2020.3.4 version). Has anyone
>>>> faced a similar issue?
>>>>
>>>> Cheers!
>>>> Lahiru
>>>>
>>>
>>>
>>> --
>>> Lahiru Jayathilake
>>> Department of Computer Science and Engineering,
>>> Faculty of Engineering,
>>> University of Moratuwa
>>>
>>> mobile : +94 (71) 912 8355
>>>
>>> <https://lk.linkedin.com/in/lahirujayathilake>
>>>
>>

Re: [apache/airavata-sandbox] Jupyter Platform Dashboard (PR #59)

[DImuthu Upeksha]

Hi Himanshu

Thanks for sending the PR. I merged your code to sandbox and it looks good.
I will gradually borrow react code into the original scigap ipython
repository [1]. I have written a java-based API for the backend instead of
flask because it is easy to implement the rest of the orchestration code
along with the API code. Moving forward, let's use this code base.

[1] https://github.com/SciGaP/iPython-Kerner-Changes
[2]
https://github.com/SciGaP/iPython-Kerner-Changes/tree/main/jupyter-platform/platform-api/src/main/java/org/apache/airavata/jupyter/api/controller
[3]
https://github.com/SciGaP/iPython-Kerner-Changes/blob/main/jupyter-platform/platform-api/src/main/java/org/apache/airavata/jupyter/core/OrchestrationEngine.java

Thanks
Dimuthu


On Sat, Jun 25, 2022 at 1:18 AM Himanshu Hansaria 
wrote:

> Added project for jupyter platform dashboard.
> --
> You can view, comment on, or merge this pull request online at:
>
>   https://github.com/apache/airavata-sandbox/pull/59
> Commit Summary
>
>- 1313430
>
> 
>First phase of development
>- d2c47de
>
> 
>Updated README and requirements.txt
>- 0ad0a70
>
> 
>Updated README
>
> File Changes
>
> (25 files )
>
>- *A* gsoc2022/jupyter-platform-dashboard/README.md
>
> 
>(37)
>- *A*
>
> gsoc2022/jupyter-platform-dashboard/backend/__pycache__/__init__.cpython-310.pyc
>
> 
>(0)
>- *A*
>gsoc2022/jupyter-platform-dashboard/backend/__pycache__/app.cpython-310.pyc
>
> 
>(0)
>- *A*
>gsoc2022/jupyter-platform-dashboard/backend/__pycache__/app.cpython-38.pyc
>
> 
>(0)
>- *A*
>
> gsoc2022/jupyter-platform-dashboard/backend/__pycache__/models.cpython-310.pyc
>
> 
>(0)
>- *A* gsoc2022/jupyter-platform-dashboard/backend/app.py
>
> 
>(111)
>- *A* gsoc2022/jupyter-platform-dashboard/backend/requirements.txt
>
> 
>(4)
>- *A* gsoc2022/jupyter-platform-dashboard/package-lock.json
>
> 
>(17058)
>- *A* gsoc2022/jupyter-platform-dashboard/package.json
>
> 
>(42)
>- *A* gsoc2022/jupyter-platform-dashboard/public/favicon.ico
>
> 
>(0)
>- *A* gsoc2022/jupyter-platform-dashboard/public/index.html
>
> 
>(43)
>- *A* gsoc2022/jupyter-platform-dashboard/public/logo192.png
>
> 
>(0)
>- *A* gsoc2022/jupyter-platform-dashboard/public/logo512.png
>
> 
>(0)
>- *A* gsoc2022/jupyter-platform-dashboard/public/manifest.json
>
> 
>(25)
>- *A* gsoc2022/jupyter-platform-dashboard/public/robots.txt
>
> 
>(3)
>- *A* gsoc2022/jupyter-platform-dashboard/src/App.css
>
> 

Re: Airavata Django Portal

[DImuthu Upeksha]

Hi Lahiru,

Thanks for the PR. I just merged it. For the error in django portal, you
need to verify whether you can telnet to airavata.host:8443. If not, there
is some issue in API Server. Please send API server logs as well

Dimuthu

On Sun, Jun 26, 2022 at 7:06 AM Lahiru Jayathilake <
lahirujayathil...@gmail.com> wrote:

> Hello Everyone,
>
> A PR was created addressing the following issues. (
> https://github.com/apache/airavata/pull/306)
> * Due to some recent security changes were done to "less secure Gmail
> accounts" on 30-May-2022[1] (Updated the
> airavata/modules/ide-integration/README.md)
> * An Enum error when running the JobMonitorStarter (updated the
> email-config.yaml file)
>
> I was following
> https://github.com/apache/airavata/tree/develop/modules/ide-integration doc
> and when I tried to log in to the portal using "default-admin/123456"
> credentials I'm getting the following error.
>
> HTTPSConnectionPool(host='airavata.host', port=8443): Max retries exceeded
>> with url: /auth/realms/default/protocol/openid-connect/token (Caused by
>> SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol
>> (_ssl.c:1122)')))
>
>
> Full log - https://dpaste.com/3MLV5CSKH
>
> Does anyone know how to resolve this?
>
> [1] - https://support.google.com/accounts/answer/6010255?hl=en
>
> Thanks,
> Lahiru
>
>
> On Fri, Jun 24, 2022 at 2:24 AM Lahiru Jayathilake <
> lahirujayathil...@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> I've been going through the following documentation to set up a local
>> Airavata Django development environment and try out the given samples.
>>
>> * https://github.com/apache/airavata/tree/develop/modules/ide-integration
>> * https://github.com/apache/airavata-django-portal#getting-started
>> *
>> https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/new_django_app/
>>
>> Due to some code changes, one of the docs needed to be updated.
>> I've already created a pull request for that -
>> https://github.com/apache/airavata-django-portal/pull/105
>>
>>
>> When starting the API server (according to this
>> 
>> guide) I encountered an issue with the IntelliJ Idea version 2022.1.3.
>> (MacOS)
>>
>> Exception - Module 'platform-monitoring' production:
>>> java.lang.NoClassDefFoundError: com/intellij/openapi/vfs/VfsUtil
>>
>>
>> Tried updating the relevant plugins and a few ways suggested in the
>> Jetbrains issue track. However, the only way I could resolve this was to
>> downgrade the Intellij Idea. (working for 2020.3.4 version). Has anyone
>> faced a similar issue?
>>
>> Cheers!
>> Lahiru
>>
>
>
> --
> Lahiru Jayathilake
> Department of Computer Science and Engineering,
> Faculty of Engineering,
> University of Moratuwa
>
> mobile : +94 (71) 912 8355
>
> 
>

Re: Repository and Front end Technology

[DImuthu Upeksha]

Hi Himanshu,

We need to understand your designs and development plan before reviewing
your code to make sure that we are on the same page. Please send a
description of how and what you are going to implement for the benefit of
others in this mail list who do not have a context

Thanks
Dimuthu

On Fri, Jun 3, 2022 at 12:19 PM Hansaria, Himanshu  wrote:

> Hello all,
>
> I'm looking to get started with the UI for the Airavata Jupyter
> application.
>
> May I know if I can continue with this on ReactJS?
>
> @De Silva, Dinuka S 
> Can you please provide me with a repository to push my codes.
>
> Thanks,
> Kind regards,
> Himanshu
>

[Airavata] Overriding and removing SLURM script parameters

[DImuthu Upeksha]

Hi All,

With the introduction of GPU queues in various clusters, traditional
SLURM scripts need several changes and those changes are not consistent
with each cluster. Because of that, we need to override SLURM script
parameters depending on the cluster configuration. This PR [1] provides the
flexibility for users to either override existing SLURM parameters or
remove them from the scripts. For example, GPU queues on Expanse do not
accept scripts with #SBATCH -n parameter and you can remove it by adding a
pre-job command RM #SBATCH -n.

I have rebased and merged this as it was extensively tested on
Ultrascan environment.

[1] https://github.com/apache/airavata/pull/277

Thanks
Dimuthu

[Airavata] Log4j migration and notes for future development

[DImuthu Upeksha]

Hi All,

As you might all have heard, some versions of log4j, including 1.x and  2.x
have been exposed to remote code execution attacks [1], and the
recommendation to avoid this is to upgrade log4j to v2.16.0. Currently,
Airavata is running on log4j 1.x, and this [2] pull request should migrate
the entire project. to 2.16.0. I have scanned all the dependencies and
excluded log4j 1.x related artifacts at the definition level. Any library
that depends on log4j 1.x now routes logs through log4j-1.2-api bridge to
log4j 2.16.0. If you are adding a new module or dependency to the project
in the future, please make sure that following steps are covered.

1. Do not add any log4j related dependency to any module. All are loaded at
the main pom level so you can use it.
2. If you are adding a new dependency, make sure that it does not include
any log4j dependency in to the project. If it does, exclude it at the
definition level. Example - [3] . You can scan derived dependencies by
running mvn dependency:tree | grep log4j
3. Always use slf4j logging API to add logs in to the code

[1] https://www.lunasec.io/docs/blog/log4j-zero-day/
[2] https://github.com/apache/airavata/pull/275/files
[3]
https://github.com/apache/airavata/pull/275/files#diff-d5149326cfe403e4106239a432c405d04be11f1588a3d566526b4ce547fcea0bR111

Thanks
Dimuthu

Re: PyPI project ownership

[DImuthu Upeksha]

Hi Marcus,

Sorry for missing the earlier invitation. I just accepted it.

Thanks
Dimuthu


On Mon, Oct 4, 2021 at 6:00 PM Christie, Marcus Aaron 
wrote:

> Looks like the invite expired. I've resent them.
>
> > On Sep 2, 2021, at 5:55 PM, Christie, Marcus Aaron 
> wrote:
> >
> > Hi Dimuthu, Isuru,
> >
> > I think it's important that PyPI projects not be owned by only a single
> Airavata committer so that updates can be pushed to PyPI even if one of us
> is away.
> >
> > To that end, I've invited you both to be co-owners of two PyPI projects
> I created:
> >
> > https://pypi.org/project/airavata-django-portal-sdk/
> > https://pypi.org/project/airavata-python-sdk/
> >
> > You should have received an invitation email.
> >
> > Likewise, can I be made a co-owner of the custos SDK on PyPI?
> >
> > Let me know if you have any questions.
> >
> >
> > Thanks,
> >
> > Marcus
>
>

Reconsidering SSH client library

[DImuthu Upeksha]

Hi Folks,

We have been using JCraft for a long time until when we move to SSHJ [3]
recently as the standard ssh client library of Airavata job manager. Main
reasons for that migration are the lack of error handlers and
implementation complexity of JCraft whereas SSHJ, provided cleanly defined
APIs to perform ssh command invocations and SFTP operations. The downside
of SSHJ is, it is mainly driven by one developer even though there are PRs
coming from other users. However, SSHJ is actively being developed (last
release Aug 2020 [4]) compared to JCraft (last release Nov 2018 [2])

Comparing both libraries, we had to manually implement connection pooling
support externally as both were designed for one-time invocation. We had to
manipulate the sessions and connections from outside to make sure that all
are utilized optimally without crashing the connections. However, from time
to time we are still seeing issues in this implementation as it is not the
natural way to go with either of these libraries.

While googling, I found Apache Mina SSHD [5]  is getting more development
attention these days [6]. Their documentation is also pretty good compared
to other libraries and as it is based on Apache Mina asynchronous IO, I
think that the performance also should be good. From the feature-wise,
session keepalive function [8] might open new ways to reuse a connection
without external pools and the remote-to-remote transfer [9] function will
release the trouble of handling input and output streams manually.

I'm going to give it a try in the next week as time permits but if any of
you have some experience working on mina-sshd library, please let me know.

[1] http://www.jcraft.com/jsch/
[2] https://mvnrepository.com/artifact/com.jcraft/jsch
[3] https://github.com/hierynomus/sshj
[4] https://mvnrepository.com/artifact/com.hierynomus/sshj/0.30.0
[5] https://github.com/apache/mina-sshd
[6] https://mvnrepository.com/artifact/org.apache.sshd/sshd-core
[7] https://github.com/apache/mina-sshd/blob/master/docs/client-setup.md
[8]
https://github.com/apache/mina-sshd/blob/master/docs/client-setup.md#keeping-the-session-alive-while-no-traffic
[9]
https://github.com/apache/mina-sshd/blob/master/docs/scp.md#remote-to-remote-transfer

Thanks
Dimuthu

Re: Data Catalog into its own repo

[DImuthu Upeksha]

+1

On Mon, Nov 9, 2020 at 11:28 AM Christie, Marcus Aaron 
wrote:

> +1
>
> > On Nov 9, 2020, at 9:35 AM, Marru, Suresh  wrote:
> >
> > Hi All,
> >
> > Airavata Experiment catalog evolved over time and though the replica
> catalog and data product models are stand alone, they are buried to use
> them outside the experiment context. Any objections to refactor experiment
> catalog and make data catalog a first class repo, in the lines of Custos
> and MFT?
> >
> > Cheers,
> > Suresh
>
>

Re: [em-center-gateway] MFT requirements for EMC portal

[DImuthu Upeksha]

Hi Dinuka,

Sorry for the late reply. Please find my reply inline. Hope that helps.
However it would be great if you can come up with a simple component /
architecture diagram that visualizes how these components might connect to
each other. It doesn't have to be perfect but atleast we might be able to
help you to figure out missing pieces.

Thanks
Dimuthu

On Tue, Oct 20, 2020 at 9:01 AM Dinuka Desilva 
wrote:

>
>
> On Mon, Oct 19, 2020 at 9:07 PM Dinuka Desilva 
> wrote:
>
>> Hi Dimuthu,
>>
>> Let me bring back the questions we discussed at the call to a discussion
>> thread. We currently see the following requirements from MFT. Let's discuss
>> further here.
>>
>> 1) Transferring files from the microscopes to the users storage. In this,
>> there's no authentication layer between the users other than a root level
>> authentication (if there's anything possible). The user and folder
>> structure has to be understood from the folder or file path specified at
>> the microscope.
>>
>> Eg: if the file path at the microscope is
>> "./abce-microscope/dinuka/session-36/bacteria-sample.png", It should be
>> uploaded to the user "dinuka" to the folder "./abce-microscope/session-36"
>> in his user drive.
>>
>> There should be another intelligent layer that should instruct MFT to do
this. You have to specify 2 resources for source and destination in MFT and
then invoke the transfer API to transfer those files.

@Marru, Suresh  @Kuruvilla, Tanya Elizabeth  I
>> would +1 for above. I believe it's good to let the users share the files to
>> the groups manually instead of taking that also from the microscope file
>> path. In my perspective, I would like not to share directly. Instead I
>> would like to go back to my computer and see the files and do any
>> modifications if needed and then share with the PI groups.
>>
>> 2) For file management, there's the need of managing folder structure for
>> individual users. And the users should be able to share the files or
>> folders to individual users or groups.
>>
>> @Christie, Marcus Aaron  I think for the user storage
>> management within the diravata-django-portal also, we could use MFT if
>> these requirements are satisfied from MFT.
>>
>
> Looking at the intention of MFT, I wonder if this is outside MFT scope. I
> believe there's no permanent storage of the files in MFT. Instead, MFT is a
> managed bridge for transferring files between two locations that could be
> from the same or different protocols.
>
>
You are correct. MFT is stateless and specialized for transferring files.
This part goes to the sharing aspect of Custos.

@isjar...@iu.edu  @Marru, Suresh 
> I was just thinking whether it's through Custos the user file storage has
> to be there. Or even as in like a seperate service providing storage for
> every Custos user.
>
>
>> 3) Form the microscopes, since there's no way of getting individual user
>> authentication processed and it has to be deeply coupled with the path of
>> the files created (1) and also since the creation of the files is
>> continuous ,there's a need of having a folder syncing capability which has
>> to keep watching the file changes.
>>
>> Regards,
>> Dinuka
>>
>>

Re: HTCondor Extension Help

[DImuthu Upeksha]

Hi Daniel,

Sorry for the late reply. Can you send me the output of following commands

docker-compose exec pga getent hosts host.docker.internal
docker-compose exec pga cat /etc/hosts
ipconfig

And the last few lines of the APIServerStarter console output.

Thanks
Dimuthu

On Tue, Oct 20, 2020 at 6:29 PM Daniel Shchur 
wrote:

> Here’s and image for reference
>
> - Daniel Shchur
>
>
>
> *From: *Daniel Shchur 
> *Sent: *Tuesday, October 20, 2020 3:36 PM
> *To: *dev@airavata.apache.org
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *RE: HTCondor Extension Help
>
>
>
> Hi Dimuthu,
>
>
>
> I must have done something wrong because a colleague of mine was able to
> get into the login page without the extra network changes with docker.
> However, he was still experiencing the issue where the login page would go
> back to the home page and note “The Airavata servers are currently down”
> when trying to login with “default-admin.” We are currently trying to go
> through the logs to see why this might be happening.
>
>
>
> Thanks!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *Daniel Shchur 
> *Sent: *Monday, October 19, 2020 9:02 PM
> *To: *dev@airavata.apache.org
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *RE: HTCondor Extension Help
>
>
>
> Hi Dimuthu,
>
>
>
> I did try the steps mentioned in [10]. I forgot to mention that when doing
> so, I wouldn’t get any connection at all. However, if I were to simply
> leave it as:
>
>
>
> docker-compose exec pga getent hosts
>
>
>
> I could then copy the last address (which wasn’t a loopback) and use the
> next `exec` function to set the container’s hosts file, allowing me to
> connect to the web interface presented by the container. I should note that
> we have one developer doing this on native Ubuntu Linux and he faced the
> same issue.
>
>
>
> Thank you!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Monday, October 19, 2020 5:29 PM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> Did you follow the steps mentioned in [10]?. I recently added couple of
> instructions that is specific to windows deployments
>
> docker-compose exec pga getent hosts host.docker.internal
>
> docker-compose exec pga /bin/sh -c "echo ' airavata.host' >> 
> /etc/hosts"
>
>
>
> [10]
> https://github.com/apache/airavata/tree/develop/modules/ide-integration#optional-starting-super-admin-portal-pga
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_tree_develop_modules_ide-2Dintegration-23optional-2Dstarting-2Dsuper-2Dadmin-2Dportal-2Dpga=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=Ypkf_DmUlrtRhSp_fDiIv6WQo6FaPTI9Qqf6Dutchw0=t9cv4PIElFkkZUmTp4QOg_j6bhsIQasdi6gKCIxrmzM=>
>
>
>
> Thanks
>
> Dimuthu
>
>
>
> On Sun, Oct 18, 2020 at 7:26 PM Daniel Shchur <
> daniel.shc...@huskers.unl.edu> wrote:
>
> Hi Dimuthu,
>
>
>
> We had some trouble initially getting the changes pulled in, but we
> eventually did and were able to run the portal. However, when running as
> recommended, the PGA portal is not able to communicate with the database
> stack with Keystore and so on, so we weren’t able to get to the login. The
> error we receive is:
>
>
>
> curl_exec() failed. Error1: Failed to connect to airavata.host port 8443:
> Connection refused
>
>
>
> One theory I had was that the containers aren’t communicating with each
> other so I added an external network within their respective docker-compose
> files might mitigate the issue, and it did. I was able to get into the
> login portal without the issue above given the resources stack and pga
> container were all on the same docker internal network. However, now when
> trying to log in, the portal redirects to the home page noting that the
> “Airavata servers are currently down”.
>
> Any help with this would be greatly appreciated. Thanks!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Wednesday, October 14, 2020 12:33 AM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> We only have tested ide integration on MacOS and Ubuntu. However I just
> managed to reproduce your issue in Windows 10 with WSL2 enabled and
> commits [5-9] done to the develop branch should fix those issues. You might
> have to merge those changes to your

Re: HTCondor Extension Help

[DImuthu Upeksha]

Hi Daniel,

Did you follow the steps mentioned in [10]?. I recently added couple of
instructions that is specific to windows deployments

docker-compose exec pga getent hosts host.docker.internal

docker-compose exec pga /bin/sh -c "echo '
airavata.host' >> /etc/hosts"


[10]
https://github.com/apache/airavata/tree/develop/modules/ide-integration#optional-starting-super-admin-portal-pga

Thanks
Dimuthu

On Sun, Oct 18, 2020 at 7:26 PM Daniel Shchur 
wrote:

> Hi Dimuthu,
>
>
>
> We had some trouble initially getting the changes pulled in, but we
> eventually did and were able to run the portal. However, when running as
> recommended, the PGA portal is not able to communicate with the database
> stack with Keystore and so on, so we weren’t able to get to the login. The
> error we receive is:
>
>
>
> curl_exec() failed. Error1: Failed to connect to airavata.host port 8443:
> Connection refused
>
>
>
> One theory I had was that the containers aren’t communicating with each
> other so I added an external network within their respective docker-compose
> files might mitigate the issue, and it did. I was able to get into the
> login portal without the issue above given the resources stack and pga
> container were all on the same docker internal network. However, now when
> trying to log in, the portal redirects to the home page noting that the
> “Airavata servers are currently down”.
>
> Any help with this would be greatly appreciated. Thanks!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Wednesday, October 14, 2020 12:33 AM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> We only have tested ide integration on MacOS and Ubuntu. However I just
> managed to reproduce your issue in Windows 10 with WSL2 enabled and
> commits [5-9] done to the develop branch should fix those issues. You might
> have to merge those changes to your fork to make it effective and I
> recommend getting a fresh clone to avoid unexpected behaviors due to temp
> files. Let us know if it works for you.
>
>
>
> [5]
> https://github.com/apache/airavata/commit/f9b9e203fc268db468977680ab511f88a3eadc99
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_commit_f9b9e203fc268db468977680ab511f88a3eadc99=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=J6UxM_Wu9xQcmoXxxvGTjmFqbH5Za77xBEF4pTynXj8=jXtAEOkym95skPFsXd9YW4PxKg8ko3fmd2lHgphKA2M=>
>
> [6]
> https://github.com/apache/airavata/commit/3794421f01fdb4ef5c0199fdfc6cfc15c25d2e7a
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_commit_3794421f01fdb4ef5c0199fdfc6cfc15c25d2e7a=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=J6UxM_Wu9xQcmoXxxvGTjmFqbH5Za77xBEF4pTynXj8=avw7ECyJ3UUo0YYlphKm8LWxDrf_wHmaoUsMSSx_CyU=>
>
> [7]
> https://github.com/apache/airavata/commit/f581b61d44eb9a848b866456acd63f29e2bbfc8e
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_commit_f581b61d44eb9a848b866456acd63f29e2bbfc8e=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=J6UxM_Wu9xQcmoXxxvGTjmFqbH5Za77xBEF4pTynXj8=_LrHNm7vMt75JMADyCyHs7VWKutPjKZCjw8YLnZF0xQ=>
>
> [8]
> https://github.com/apache/airavata/commit/9e30529d7545a1df0b9d94c2783d7d11c32d301d
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_commit_9e30529d7545a1df0b9d94c2783d7d11c32d301d=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=J6UxM_Wu9xQcmoXxxvGTjmFqbH5Za77xBEF4pTynXj8=aO43YTRpZ8PLuZ84xcB_yR-5yT8GOKXtfwHKxAgdkak=>
>
> [9]
> https://github.com/apache/airavata/commit/b11d81182f915491187dcf9f3ede5beeb5ea94a2
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_commit_b11d81182f915491187dcf9f3ede5beeb5ea94a2=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=J6UxM_Wu9xQcmoXxxvGTjmFqbH5Za77xBEF4pTynXj8=k_GXTpaj3wcjCgsjj8PIzxxFISaGmYekMoTg2rk5aM0=>
>
>
>
> Thanks
>
> Dimuthu
>
>
>
> On Tue, Oct 13, 2020 at 6:37 PM Daniel Shchur <
> daniel.shc...@huskers.unl.edu> wrote:
>
> Hi Dimuthu,
>
>
>
> Windows 10 2004 19041.508
>
> Docker v19.03.13 running on WSL2 - Ubuntu 20.04
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Tuesday, October 13, 2020 5:34 PM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> 

Re: HTCondor Extension Help

[DImuthu Upeksha]

Hi Daniel,

We only have tested ide integration on MacOS and Ubuntu. However I just
managed to reproduce your issue in Windows 10 with WSL2 enabled and
commits [5-9] done to the develop branch should fix those issues. You might
have to merge those changes to your fork to make it effective and I
recommend getting a fresh clone to avoid unexpected behaviors due to temp
files. Let us know if it works for you.

[5]
https://github.com/apache/airavata/commit/f9b9e203fc268db468977680ab511f88a3eadc99
[6]
https://github.com/apache/airavata/commit/3794421f01fdb4ef5c0199fdfc6cfc15c25d2e7a
[7]
https://github.com/apache/airavata/commit/f581b61d44eb9a848b866456acd63f29e2bbfc8e
[8]
https://github.com/apache/airavata/commit/9e30529d7545a1df0b9d94c2783d7d11c32d301d
[9]
https://github.com/apache/airavata/commit/b11d81182f915491187dcf9f3ede5beeb5ea94a2

Thanks
Dimuthu

On Tue, Oct 13, 2020 at 6:37 PM Daniel Shchur 
wrote:

> Hi Dimuthu,
>
>
>
> Windows 10 2004 19041.508
>
> Docker v19.03.13 running on WSL2 - Ubuntu 20.04
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Tuesday, October 13, 2020 5:34 PM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> I'm trying to reproduce the error locally. Can you please send me the
> following information of your local setup.
>
>
>
> OS / version
>
> Docker version
>
>
>
> Thanks
>
> Dimuthu
>
>
>
> On Tue, Oct 13, 2020 at 5:32 PM Daniel Shchur <
> daniel.shc...@huskers.unl.edu> wrote:
>
> Hi Dimunth,
>
>
>
> We have been trying to get the PGA running; however, it seems to fail
> building currently during the optimize portion. This is after having
> started the resources stack and starting the JobEngineStarter and
> AiravataAPI. Those parts start up without a hitch. I also added in the
> hosts file changes.
>
>
>
> The error is:
>
>
>
> > php artisan optimize
>
> Generating optimized class loader
>
> Compiling common classes
>
> Compiling views
>
>
>
>
>
>
>   [ErrorException]
>
>
>   
> file_put_contents(/var/www/portals/default/app/storage/views/16ac8ea2e3f88487e87226a07d300af0):
> failed to open stream: No such file or directory
>
>
>
>
>
>
> optimize [--force] [--psr]
>
>
>
> Script php artisan optimize handling the post-install-cmd event returned
> with error code 1
>
> Any help would be greatly appreciated. Thanks!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Friday, October 9, 2020 8:46 AM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> Please send you PR to airavata:htcondor branch which I created yesterday.
> Then I also can help you to speed up the development process.
>
>
>
> Thanks
>
> Dimuthu
>
>
>
> On Thu, Oct 8, 2020 at 5:59 PM Daniel Shchur <
> daniel.shc...@huskers.unl.edu> wrote:
>
> Hi Dimuthu,
>
>
>
> That clears up a lot of confusion. Thank you!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Thursday, October 8, 2020 4:58 PM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> That's correct, we assume all job manager configurations have unique set
> of command types like job submission, cancellation, status checking etc.
> It's up to the end user to specify the exact command for each command type.
> For example, SLURM job submission command is sbatch. So the user has to
> specify that though the portal which will eventually be stored in the
> database.
>
>
>
> Thanks
>
> Dimuthu
>
>
>
>
>
> On Thu, Oct 8, 2020 at 5:14 PM Daniel Shchur <
> daniel.shc...@huskers.unl.edu> wrote:
>
> Hi Dimuthu,
>
>
>
> So, to be clear,
>
>
>
> The relevant job submission commands are registered at the client-side? As
> in, the front end interface and are not hardcoded into the backend?
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Thursday, October 8, 2020 4:12 PM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> If you look at [2], it's where the protocol specific configuration object
> is created. Each implementation is a subclass of the
> JobManagerConfiguration inte

Re: HTCondor Extension Help

[DImuthu Upeksha]

Hi Daniel,

I'm trying to reproduce the error locally. Can you please send me the
following information of your local setup.

OS / version
Docker version

Thanks
Dimuthu

On Tue, Oct 13, 2020 at 5:32 PM Daniel Shchur 
wrote:

> Hi Dimunth,
>
>
>
> We have been trying to get the PGA running; however, it seems to fail
> building currently during the optimize portion. This is after having
> started the resources stack and starting the JobEngineStarter and
> AiravataAPI. Those parts start up without a hitch. I also added in the
> hosts file changes.
>
>
>
> The error is:
>
>
>
> > php artisan optimize
>
> Generating optimized class loader
>
> Compiling common classes
>
> Compiling views
>
>
>
>
>
>
>   [ErrorException]
>
>
>   
> file_put_contents(/var/www/portals/default/app/storage/views/16ac8ea2e3f88487e87226a07d300af0):
> failed to open stream: No such file or directory
>
>
>
>
>
>
> optimize [--force] [--psr]
>
>
>
> Script php artisan optimize handling the post-install-cmd event returned
> with error code 1
>
> Any help would be greatly appreciated. Thanks!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Friday, October 9, 2020 8:46 AM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> Please send you PR to airavata:htcondor branch which I created yesterday.
> Then I also can help you to speed up the development process.
>
>
>
> Thanks
>
> Dimuthu
>
>
>
> On Thu, Oct 8, 2020 at 5:59 PM Daniel Shchur <
> daniel.shc...@huskers.unl.edu> wrote:
>
> Hi Dimuthu,
>
>
>
> That clears up a lot of confusion. Thank you!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Thursday, October 8, 2020 4:58 PM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> That's correct, we assume all job manager configurations have unique set
> of command types like job submission, cancellation, status checking etc.
> It's up to the end user to specify the exact command for each command type.
> For example, SLURM job submission command is sbatch. So the user has to
> specify that though the portal which will eventually be stored in the
> database.
>
>
>
> Thanks
>
> Dimuthu
>
>
>
>
>
> On Thu, Oct 8, 2020 at 5:14 PM Daniel Shchur <
> daniel.shc...@huskers.unl.edu> wrote:
>
> Hi Dimuthu,
>
>
>
> So, to be clear,
>
>
>
> The relevant job submission commands are registered at the client-side? As
> in, the front end interface and are not hardcoded into the backend?
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Thursday, October 8, 2020 4:12 PM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> If you look at [2], it's where the protocol specific configuration object
> is created. Each implementation is a subclass of the
> JobManagerConfiguration interface [3]. So each implementation might have
> different execution commands and those are stored in the database and you
> can register them through PGA. Please look at the screenshot I sent
> earlier. Once helix figured out the right Configuration implementation, it
> invokes it at the [4]. Please let me know if this is clear
>
>
>
> [2]
> https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobFactory.java#L106
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_develop_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_config_JobFactory.java-23L106=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=TIv6Hm841JVIH_TucpIxiNWYVrZ9o8BO-aAKlXeLNCU=jOokB292No_5nhnM9IQCJ3f4HgYCBysW7cYy7r_iwns=>
>
> [3]
> https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobManagerConfiguration.java
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_develop_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_config_JobManagerConfiguration.java=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMty

Re: HTCondor Extension Help

[DImuthu Upeksha]

Hi Daniel,

Please send you PR to airavata:htcondor branch which I created yesterday.
Then I also can help you to speed up the development process.

Thanks
Dimuthu

On Thu, Oct 8, 2020 at 5:59 PM Daniel Shchur 
wrote:

> Hi Dimuthu,
>
>
>
> That clears up a lot of confusion. Thank you!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Thursday, October 8, 2020 4:58 PM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> That's correct, we assume all job manager configurations have unique set
> of command types like job submission, cancellation, status checking etc.
> It's up to the end user to specify the exact command for each command type.
> For example, SLURM job submission command is sbatch. So the user has to
> specify that though the portal which will eventually be stored in the
> database.
>
>
>
> Thanks
>
> Dimuthu
>
>
>
>
>
> On Thu, Oct 8, 2020 at 5:14 PM Daniel Shchur <
> daniel.shc...@huskers.unl.edu> wrote:
>
> Hi Dimuthu,
>
>
>
> So, to be clear,
>
>
>
> The relevant job submission commands are registered at the client-side? As
> in, the front end interface and are not hardcoded into the backend?
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Thursday, October 8, 2020 4:12 PM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> If you look at [2], it's where the protocol specific configuration object
> is created. Each implementation is a subclass of the
> JobManagerConfiguration interface [3]. So each implementation might have
> different execution commands and those are stored in the database and you
> can register them through PGA. Please look at the screenshot I sent
> earlier. Once helix figured out the right Configuration implementation, it
> invokes it at the [4]. Please let me know if this is clear
>
>
>
> [2]
> https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobFactory.java#L106
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_develop_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_config_JobFactory.java-23L106=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=TIv6Hm841JVIH_TucpIxiNWYVrZ9o8BO-aAKlXeLNCU=jOokB292No_5nhnM9IQCJ3f4HgYCBysW7cYy7r_iwns=>
>
> [3]
> https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobManagerConfiguration.java
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_develop_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_config_JobManagerConfiguration.java=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=TIv6Hm841JVIH_TucpIxiNWYVrZ9o8BO-aAKlXeLNCU=E9HjTKXLL3YhkdLhA98G3r0fH7lpD5WKlJldHoFI32E=>
>
> [4]
> https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/JobSubmissionTask.java#L78
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_develop_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_JobSubmissionTask.java-23L78=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=TIv6Hm841JVIH_TucpIxiNWYVrZ9o8BO-aAKlXeLNCU=yuHKbjJZT93zcEiQHAIPBNZMq6aKiKLmzkSep5i2-j8=>
>
>
>
> Thanks
>
> Dimuthu
>
>
>
> On Thu, Oct 8, 2020 at 4:58 PM Daniel Shchur <
> daniel.shc...@huskers.unl.edu> wrote:
>
> Hi all,
>
>
>
> We have been trying to figure out where the helix system actually
> specifies job submission protocol commands (e.g. sbatch, squeue, scancel).
> The only direct references we could find are part of the client-sdk
> “registerSampleApplication”. Everything else, when referencing
> “ResourceJobConfiguration” seems to be fairly circular, or goes to dead-end
> files that may not be used anymore.
>
>
>
> Question: is there a direct command mapping for the
> “Map” that is called within the code, or is this
> set elsewhere? We haven’t found any references to where Airavata would know
> how to submit a SLURM job so that we can let it know how to submit an
>

Re: HTCondor Extension Help

[DImuthu Upeksha]

Hi Daniel,

If you look at [2], it's where the protocol specific configuration object
is created. Each implementation is a subclass of the
JobManagerConfiguration interface [3]. So each implementation might have
different execution commands and those are stored in the database and you
can register them through PGA. Please look at the screenshot I sent
earlier. Once helix figured out the right Configuration implementation, it
invokes it at the [4]. Please let me know if this is clear

[2]
https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobFactory.java#L106
[3]
https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobManagerConfiguration.java
[4]
https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/JobSubmissionTask.java#L78

Thanks
Dimuthu

On Thu, Oct 8, 2020 at 4:58 PM Daniel Shchur 
wrote:

> Hi all,
>
>
>
> We have been trying to figure out where the helix system actually
> specifies job submission protocol commands (e.g. sbatch, squeue, scancel).
> The only direct references we could find are part of the client-sdk
> “registerSampleApplication”. Everything else, when referencing
> “ResourceJobConfiguration” seems to be fairly circular, or goes to dead-end
> files that may not be used anymore.
>
>
>
> Question: is there a direct command mapping for the
> “Map” that is called within the code, or is this
> set elsewhere? We haven’t found any references to where Airavata would know
> how to submit a SLURM job so that we can let it know how to submit an
> HTCondor job.
>
> Any help on this would be greatly appreciated. Thanks!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Monday, October 5, 2020 9:50 AM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Sorry I didn't send the image I mentioned earlier
>
>
>
>
>
>
>
> On Sun, Oct 4, 2020 at 10:51 PM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com> wrote:
>
> Hi Daniel,
>
>
>
> Your PR looks good to me. I think the best place for you to start is by
> setting up the PGA portal (the old portal we used before the django one)
> and adding a resource manager type for the compute resource as you can see
> in the attached image. You can set up the pga locally using this [1] guide.
> Once you can register the resource manager type using the pga portal, you
> can use the django portal to submit jobs. I believe that you don't need to
> change any code in the django portal.
>
>
>
> [1]
> https://github.com/apache/airavata/tree/master/modules/ide-integration#optional-starting-super-admin-portal-pga
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_tree_master_modules_ide-2Dintegration-23optional-2Dstarting-2Dsuper-2Dadmin-2Dportal-2Dpga=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=tuYZSoSxpr0vpOY_QMnBJNwCbUSViATLYySXRyD33aw=8A5-Br28vywXcU42WH9bl0Ad0b5YMnXNNIplSl3r9J8=>
>
>
>
> Thanks
>
> Dimuthu
>
>
>
> On Sun, Oct 4, 2020 at 6:12 PM Daniel Shchur <
> daniel.shc...@huskers.unl.edu> wrote:
>
> Hi Dimuthu and Suresh,
>
>
>
> Thanks for getting back to us. We went ahead and made a work in progress pull
> request
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_pull_263=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=tuYZSoSxpr0vpOY_QMnBJNwCbUSViATLYySXRyD33aw=axraS0UJlcrcJlRwGIeswZKPC0cHEDu13ADapD1Xz5A=>
> for our changes we have created so far. We also took a look at the
> job-monitor section and will work towards creating additions to support
> HTCondor.
>
>
>
> Do any of you have any suggestions on what the best way is to test
> submissions to HTCondor? We looked into using the Airavata API and the
> Django portal and are curious what you would recommend, and how to do it
> using your recommendation.
>
>
>
> Thanks!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Friday, October 2, 2020 5:22 PM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> This is great progress. I think you are on the right track. Did you manage
> to set up the ide integration for the development of Airavata. If you have
> already done that, you can test these fe

Re: HTCondor Extension Help

[DImuthu Upeksha]

Hi Daniel,

Your PR looks good to me. I think the best place for you to start is by
setting up the PGA portal (the old portal we used before the django one)
and adding a resource manager type for the compute resource as you can see
in the attached image. You can set up the pga locally using this [1] guide.
Once you can register the resource manager type using the pga portal, you
can use the django portal to submit jobs. I believe that you don't need to
change any code in the django portal.

[1]
https://github.com/apache/airavata/tree/master/modules/ide-integration#optional-starting-super-admin-portal-pga

Thanks
Dimuthu

On Sun, Oct 4, 2020 at 6:12 PM Daniel Shchur 
wrote:

> Hi Dimuthu and Suresh,
>
>
>
> Thanks for getting back to us. We went ahead and made a work in progress pull
> request <https://github.com/apache/airavata/pull/263> for our changes we
> have created so far. We also took a look at the job-monitor section and
> will work towards creating additions to support HTCondor.
>
>
>
> Do any of you have any suggestions on what the best way is to test
> submissions to HTCondor? We looked into using the Airavata API and the
> Django portal and are curious what you would recommend, and how to do it
> using your recommendation.
>
>
>
> Thanks!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Friday, October 2, 2020 5:22 PM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> This is great progress. I think you are on the right track. Did you manage
> to set up the ide integration for the development of Airavata. If you have
> already done that, you can test these features. However I recommend you to
> send pull requests to a separate branch (htcondor?) and then I can provide
> you feedback on the next steps.
>
>
>
> Thanks
>
> Dimuthu
>
>
>
>
>
> On Thu, Oct 1, 2020 at 4:57 PM Suresh Marru  wrote:
>
> Hi Daniel,
>
>
>
> Thanks for this nice summary for keeping the dev list apprised of your
> progress.
>
>
>
> One additional item is to discuss and plan on job monitoring approaches,
> you can look at current references here -
> https://github.com/apache/airavata/tree/master/modules/job-monitor
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_tree_master_modules_job-2Dmonitor=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=lmrRGQ0q0u8q8eCyoDDHtDCxKniLMvrktlVw7qnV3Fo=zpJDdIRgv9hm0QkGZuHhDOYvc1_zMhUmH9TF7mx9A-0=>
>
>
>
>
> Suresh
>
>
>
> On Oct 1, 2020, at 4:52 PM, Daniel Shchur 
> wrote:
>
>
>
> Hi all,
>
>
>
> We are working on the HTCondor extension for Airavata. We have gone
> through the code and started to implement additions based on the other job
> workflow systems such as SLURM. We forked
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_docquantum_airavata_tree_htcondor=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=lmrRGQ0q0u8q8eCyoDDHtDCxKniLMvrktlVw7qnV3Fo=14ShokmbcQHBZlmV67fTrNoK8ARktUQERu6wRQQTlWc=>
>  the
> project and created a new branch (htcondor) to host our changes. We will
> make a pull request into the main Airavata repository for it once we have
> finished the implementation. A relevant issue has also been created here
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_AIRAVATA-2D3377=DwMFaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es=lmrRGQ0q0u8q8eCyoDDHtDCxKniLMvrktlVw7qnV3Fo=ZjE4skBTqEHdJqLBat2-uLoS1w6_Ksh_c5d4VPKjnS8=>
> .
>
>
>
> We would like help to know exactly where additions need to be made in case
> we missed a few spots. At this point, we know that:
>
>- A groovy template needs to be made (finished)
>- A job configuration class needs to be made (WIP)
>- A job parser class needs to be made (WIP)
>- And enums need to be modified throughout the project so that systems
>know of HTCondor as an option. (finished)
>
>
>
> If any of you could direct us to the right places, that would be greatly
> appreciated.
>
>
>
> Also, if there’s more documentation or information about the code, such as
> how the command map should work or should be implemented (configuration
> class?), that would be appreciated.
>
>
>
> Thank you!
>
>
>
> - Daniel Shchur
>
>
>
>
>

Re: HTCondor Extension Help

[DImuthu Upeksha]

Hi Daniel,

This is great progress. I think you are on the right track. Did you manage
to set up the ide integration for the development of Airavata. If you have
already done that, you can test these features. However I recommend you to
send pull requests to a separate branch (htcondor?) and then I can provide
you feedback on the next steps.

Thanks
Dimuthu


On Thu, Oct 1, 2020 at 4:57 PM Suresh Marru  wrote:

> Hi Daniel,
>
> Thanks for this nice summary for keeping the dev list apprised of your
> progress.
>
> One additional item is to discuss and plan on job monitoring approaches,
> you can look at current references here -
> https://github.com/apache/airavata/tree/master/modules/job-monitor
>
> Suresh
>
> On Oct 1, 2020, at 4:52 PM, Daniel Shchur 
> wrote:
>
> Hi all,
>
> We are working on the HTCondor extension for Airavata. We have gone
> through the code and started to implement additions based on the other job
> workflow systems such as SLURM. We forked
>  the project and
> created a new branch (htcondor) to host our changes. We will make a pull
> request into the main Airavata repository for it once we have finished the
> implementation. A relevant issue has also been created here
> .
>
> We would like help to know exactly where additions need to be made in case
> we missed a few spots. At this point, we know that:
>
>- A groovy template needs to be made (finished)
>- A job configuration class needs to be made (WIP)
>- A job parser class needs to be made (WIP)
>- And enums need to be modified throughout the project so that systems
>know of HTCondor as an option. (finished)
>
>
> If any of you could direct us to the right places, that would be greatly
> appreciated.
>
> Also, if there’s more documentation or information about the code, such as
> how the command map should work or should be implemented (configuration
> class?), that would be appreciated.
>
> Thank you!
>
> - Daniel Shchur
>
>
>

Re: Recursive entity fetching in registry

[DImuthu Upeksha]

Hi Suresh,

I believe this is more of a bug in JPA or we are configuring it in the
wrong way. If we have configured a ManyToOne relationship to a LAZY
initialization [8], it should not load ExperimentEntity if we haven't
invoked getExperiment().get.. methods. But in this case, it's automatically
initializing ExperimentEntity and that's why we are getting this recursive
object creation. What we need to do is to stop this recursive object
creation in bidirectional relationships.

As a general observation and a reply to suresh's suggestion, In the long
run, observing the whole database stack from the top level, I think we need
to consider using some sophisticated framework like spring (spring boot) to
handle JPA transactions. Currently we are coding very close to entity
managers and that might cause unexpected issues if we haven't done it
properly. That (Spring) might facilitate us to either go with GraphQL or
Grpc based on the evaluations we get.

[8]
https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentOutputEntity.java#L80

Thanks
Dimuthu

On Tue, Sep 22, 2020 at 12:15 PM Suresh Marru  wrote:

> Hi Dimuthu,
>
> I have been wondering if GraphQL will be worthwhile to explore for such
> bulk retrieval’s? I think this will be a low hanging fruit for us to try to
> integrating with something like Hasura[4] it is dominantly used for
> Postgres but they have been actively supporting MySql as well [5]
>
> I am not sure if this will address you point out though, since GraphQL
> spec doesn’t support recursive nested objects [6], simple one to many seems
> to be supported [7] but not bi directional.
>
> Suresh
>
> [4] - https://github.com/hasura/graphql-engine
> [5] - https://hasura.io/docs/1.0/graphql/core/guides/mysql-preview.html
> [6] - https://github.com/graphql/graphql-spec/issues/91
> [7] -
> https://hasura.io/docs/1.0/graphql/core/schema/table-relationships/database-modelling/one-to-many.html
>
>
> On Sep 22, 2020, at 11:01 AM, DImuthu Upeksha 
> wrote:
>
> Hi Folks,
>
> I just noticed that the entity relations that have
> bidirectional definitions (OneToMany and ManyToOne) between 2 tables cause
> recursive object fetching and it is causing a significant performance drop
> in database querying. For example [1] and [2]. When we fetch an
> experiment output, I can see a stack of experiment -> experiment outputs ->
> experiment 
>
> Same issue was mentioned in [3] but the solution is for spring boot json
> parsing. As we don't use spring boot, we might have to figure out some
> other way to stop this recursive stack creation.
>
> [1]
> https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentEntity.java#L88
> [2]
> https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentOutputEntity.java#L80
> [3]
> https://medium.com/@udith.indrakantha/issue-related-with-infinite-recursive-fetching-of-data-from-relationships-between-entity-classes-ffc5fac6c816
>
> Thanks
> Dimuthu
>
>
>

Recursive entity fetching in registry

[DImuthu Upeksha]

Hi Folks,

I just noticed that the entity relations that have
bidirectional definitions (OneToMany and ManyToOne) between 2 tables cause
recursive object fetching and it is causing a significant performance drop
in database querying. For example [1] and [2]. When we fetch an
experiment output, I can see a stack of experiment -> experiment outputs ->
experiment 

Same issue was mentioned in [3] but the solution is for spring boot json
parsing. As we don't use spring boot, we might have to figure out some
other way to stop this recursive stack creation.

[1]
https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentEntity.java#L88
[2]
https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentOutputEntity.java#L80
[3]
https://medium.com/@udith.indrakantha/issue-related-with-infinite-recursive-fetching-of-data-from-relationships-between-entity-classes-ffc5fac6c816

Thanks
Dimuthu

Re: Establishing an SSH connection to StorageResource

[DImuthu Upeksha]

Hi Vivek,

The error you are getting is self explanatory. You are trying to SSH into a
storage resource with a ssh key that is not available in the storage's
authorized_keys file. I can see that you are trying to create a new ssh
credential.
token = generateAndRegisterSSHKeys(authzToken, "");
Issue is, the public key of this credential is not copied to the
authorized_keys file of the target storage resource. You need to fetch the
SSH credential from the storage preference as we are doing in Helix [1]
[2]. These keys are already registered in the storage endpoint.

[1]
https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/staging/DataStagingTask.java#L83
[2]
https://github.com/apache/airavata/blob/7a9b27aac395883afcf9f5bf08a3eb4c6ff577a5/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/TaskContext.java#L477

Thanks
Dimuthu

On Wed, Aug 19, 2020 at 4:06 AM Bandaru, Vivek Shresta 
wrote:

> Hi All,
>
>
>
> Though the previous approach worked, it had a problem. Every gateway would
> need to implement their own user storage tracking mechanism.  As discussed
> earlier in my previous mail, the better approach would be to establish an
> SSH connection with the Storage Resource and track the storage size in
> Airavata. After a little debugging and discussions, I found that
> StorageResourceAdaptor and AgentAdaptor’s (which uses pooling) are the way
> to establish a connection and execute commands on remote machines instead
> of creating a new SSHConnection, so my new implementation uses
> StorageResourceAdaptor to establish a connection and execute the du
> 
>  commands
> to get the size of the user directory.
>
>
>
> Questions:
>
>
>
>1. Currently, in case of Airavata-Django-Portal, experimentDataDir is
>always inside the project directiory which in turn is inside the user
>directory. Will the same behavior be followed across all the gateways?
>2. I was trying to establish a connection with the gateway using
>StorageResourceAdaptor (I believe AgentAdaptor’s are used for compute
>resources) but I’m stuck at this step. I was successfully able to create a
>connection using AgentAdaptor (only for testing purposes) and execute
>commands on it. But I’m not able to do the same with
>StorageResourceAdaptor.
>The only difference I can think of is, here I’m creating a new
>authToken for establishing a connection with the Storage resource (and of
>course deleting it at the end of execution), and I’m assuming this might be
>the issue. While testing with AgentAdaptor, I hardcoded the authToken value
>that’s already generated for the Jetstream compute resources, so I’m
>guessing that might be the reason why it was working in this case.
>
>This is the error stacktrace that I’ve encountered while executing the
>‘executeCommand()’ on StorageResourceAdaptor after initiating it.
>
>
> 2020-08-18 19:31:07,152 [SSH-Pool-Monitor-airavata.host-2] INFO
> o.a.a.h.a.PoolingSSHJClient Current active connections for  null @
> airavata.host : 2 are 0 []
>
> 2020-08-18 19:31:07,173 [pool-24-thread-6] INFO
> n.s.s.transport.TransportImpl Client identity string: SSH-2.0-SSHJ_0.23.0 []
>
> 2020-08-18 19:31:07,180 [pool-24-thread-6] INFO
> n.s.s.transport.TransportImpl Server identity string: SSH-2.0-OpenSSH_7.2p2
> Ubuntu-4ubuntu2.7 []
>
> 2020-08-18 19:31:07,488 [pool-24-thread-6] ERROR
> o.a.a.a.s.h.AiravataServerHandler
> net.schmizz.sshj.userauth.UserAuthException: Exhausted available
> authentication methods []
>
> org.apache.airavata.agents.api.AgentException:
> net.schmizz.sshj.userauth.UserAuthException: Exhausted available
> authentication methods
>
> at
> org.apache.airavata.helix.adaptor.SSHJAgentAdaptor.executeCommand(SSHJAgentAdaptor.java:190)
>
> at
> org.apache.airavata.helix.adaptor.SSHJStorageAdaptor.executeCommand(SSHJStorageAdaptor.java:94)
>
> at
> org.apache.airavata.api.server.handler.AiravataServerHandler.validateStorageLimit(AiravataServerHandler.java:726)
>
> at
> org.apache.airavata.api.Airavata$Processor$validateStorageLimit.getResult(Airavata.java:18296)
>
> at
> org.apache.airavata.api.Airavata$Processor$validateStorageLimit.getResult(Airavata.java:18280)
>
> at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>
> at
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> 

Re: Creating experiments on local

[DImuthu Upeksha]

Hi Vivek,

Yes instructions should be updated in the wiki page. However, you can
follow the instructions provided in [1] to deploy Airavata in the IDE. It
should be upto date.

[1] https://github.com/apache/airavata/tree/master/modules/ide-integration

Thanks
Dimuthu

On Thu, Aug 6, 2020 at 3:19 AM Bandaru, Vivek Shresta 
wrote:

> Thanks Marcus, seems like that’s the issue. APIServerStarter.java doesn’t
> initiate JobEngineStarter.java. Weirdly the Airavata’s Local setup
> document
> 
> doesn’t talk about JobEngineStarter.java.
>
> I see that JobMonitorStarter.java(Looks like it’s an email module which
> doesn’t interfere with Airavata’s functionality) is also never initiated.
> Are there any such files which I should know about which should be
> initiated along with APIServerStarter.java and JobEngineStarter.java ?
>
>
>
> *From: *"Christie, Marcus Aaron" 
> *Reply-To: *
> *Date: *Monday, August 3, 2020 at 11:23 AM
> *To: *Airavata Dev 
> *Subject: *Re: Creating experiments on local
>
>
>
> Hi Vivek,
>
>
>
> Check the output of JobEngineStarter. Sounds like Helix is down if the
> experiment is stuck in Launched state.
>
>
>
> On Jul 24, 2020, at 3:12 AM, Bandaru, Vivek Shresta 
> wrote:
>
>
>
> Gentle reminder. I’m still looking for inputs.
>
>
>
> *From: *"Bandaru, Vivek Shresta" 
> *Date: *Monday, July 20, 2020 at 6:14 PM
> *To: *"dev@airavata.apache.org" 
> *Subject: *Creating experiments on local
>
>
>
> Hi All,
>
>
>
>- I’ve been trying to understand how a user experiment’s input/output
>storage works. Currently the only experiment which I can create in my local
>is the application ‘Echo’ which doesn’t require any user input file and for
>some reason is never completing and is always in the Launched state.
>In the below screenshot, you can see experiments which have been running
>since 3 days. What can the issue be?
>
>
>
> 
>
>
>
> The below screenshot shows that I’ve chosen the default resources for
> computing the experiment.
>
>
>
> 
>
>
>
>
>
>- Executing an experiment which requires an input  file and generated
>an output file will definitely help me understand the process better. The
>only experiment available in my local is ‘Echo’ which doesn’t generate any
>output. How do I launch any other experiments which are only available on
>https://testdrive.airavata.org/?
>
>
>
> Regards,
>
> Vivek.
>
>
>

Re: Updating the Airavata README.md

[DImuthu Upeksha]

Hi Marcus,

This is very nice and clean. I suggested a minor change and what do you
think if we point to ide integration module in the readme as a quick start
point?

Thanks
Dimuthu

On Wed, Aug 5, 2020 at 11:01 AM Christie, Marcus Aaron 
wrote:

> Hi All,
>
> I've attempted to bring the Airavata README.md up to date. It contained
> some stale information, like project structure, which I didn't bother
> trying to update. I added some additional links and information, based on
> typical things I've seen on other READMEs (like, for example, how to
> contribute).
>
> https://github.com/apache/airavata/pull/258
>
> Please review the PR when you have a chance. If there is anything else you
> think should be in the README just let me know, I'd be happy to add it.
>
> Thanks,
>
> Marcus

Re: [External] Re: Airavata MFT- Gdrive transport implementation

[DImuthu Upeksha]

Hi Shivali,

This looks great. This is a good habit for other transport developers also.
Once we have some quality content, we can directly put into the repository
wiki. I requested some changes to your PR and please update relevant parts
of the document once you addressed them.

Thanks
Dimuthu

On Fri, May 1, 2020 at 1:33 PM Jejurkar, Shivali  wrote:

> Hello,
>
>
>
> We have created a document for setting up service account for Google Drive
> transport setup in MFT. Kindly review  and share your comments/feedback.
>
>
> https://docs.google.com/document/d/1ql3megjxd_4_t9ID23se9Lb8NFpY83e175XHoKYpYdg/edit?usp=sharing
>
>
>
>
>
> Thanks,
>
> Shivali Jejurkar
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> *From: *Bafna, Nikita 
> *Sent: *Thursday, April 30, 2020 12:43 PM
> *To: *dev@airavata.apache.org
> *Subject: *RE: [External] Re: Airavata MFT- Gdrive transport
> implementation
>
>
>
> Hi Dimuthu,
>
>
>
> We have now addressed all the issues raised.
>
> Could you please verify and provide your inputs.
>
>
>
> Regards,
>
> Nikita
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Wednesday, April 29, 2020 1:49 AM
> *To: *Airavata Dev 
> *Subject: *Re: [External] Re: Airavata MFT- Gdrive transport
> implementation
>
>
> Still there are some unaddressed reviews available. Please have a look at
> them.
>
> [2] https://github.com/apache/airavata-mft/pull/11/files
>
> On Wed, Apr 29, 2020 at 1:33 AM Patil, Aniruddha Vivek 
> wrote:
>
>> Hello Dimuthu,
>>
>> We addressed the changes suggested in your review. Please let us know
>> what should be addressed further.
>>
>> Thanks and regards,
>> Aniruddha Patil
>>
>> On Apr 28, 2020, at 10:05 PM, DImuthu Upeksha 
>> wrote:
>>
>> Thanks. I did the first pass of you PR. Please have a look. Once you
>> address the issues, I will do more detailed reviews.
>>
>> Dimuthu
>>
>> On Tue, Apr 28, 2020 at 6:22 PM Bafna, Nikita  wrote:
>>
>>> Hello Dimuthu,
>>>
>>>
>>> Sorry I forgot to mention the PR [1] link. Here it is.
>>>
>>>
>>> [1] https://github.com/apache/airavata-mft/pull/11
>>>
>>>
>>> Regards,
>>>
>>> Nikita
>>>
>>>
>>> *From: *Bafna, Nikita 
>>> *Sent: *Tuesday, April 28, 2020 4:39 PM
>>> *To: *dev@airavata.apache.org
>>> *Subject: *Re: [External] Re: Airavata MFT- Gdrive transport
>>> implementation
>>>
>>>
>>> Hello Dimuthu,
>>>
>>> We have implemented transport system for Google drive.
>>>
>>> We have build our transport in a way where if a file with that name
>>> already exists it will update that file instead of creating a new one on
>>> Google drive.
>>>
>>> For now we have tested our implementation between endpoints below.
>>>
>>> S3 to Google drive
>>>
>>> Google drive to S3
>>>
>>> Local to Google drive (Screenshot attached)
>>>
>>> Google drive to Local (Screenshot attached)​
>>>
>>> Could you please have a look and let us know your feedback.
>>>
>>> Thank you,
>>> Team NSA
>>> (Nikita Bafna, Shivali Jejurkar, Aniruddha Patil)
>>>
>>>
>>> --
>>> *From:* Bafna, Nikita
>>> *Sent:* Tuesday, April 28, 2020 5:20 AM
>>> *To:* dev@airavata.apache.org
>>> *Subject:* Re: [External] Re: Airavata MFT- Gdrive transport
>>> implementation
>>>
>>>
>>> Hi Dimuthu,
>>>
>>>
>>> During testing of the GDrive transport, we came across a situation where
>>> if a file is being uploaded on the Gdrive and if the file with same name
>>> already exists, then Gdrive creates a new file with the same name (it
>>> creates a new version ). So how should this be handled? Should it create a
>>> new file or update the existing file?
>>>
>>>
>>> Let us know your inputs.
>>>
>>>
>>> Regards,
>>>
>>> Nikita
>>> --
>>> *From:* Bafna, Nikita 
>>> *Sent:* Tuesday, April 28, 2020 12:48 AM
>>> *To:* dev@airavata.apache.org
>>> *Subject:* RE: [External] Re: Airavata MFT- Gdrive transport
>>> implementation
>>>
>>>
>>> Okay.
>>>
>>> Thanks for the inputs Dimuthu.
>>>
>&

Re: [External] Re: Airavata MFT- Gdrive transport implementation

[DImuthu Upeksha]

Still there are some unaddressed reviews available. Please have a look at
them.

[2] https://github.com/apache/airavata-mft/pull/11/files

On Wed, Apr 29, 2020 at 1:33 AM Patil, Aniruddha Vivek 
wrote:

> Hello Dimuthu,
>
> We addressed the changes suggested in your review. Please let us know what
> should be addressed further.
>
> Thanks and regards,
> Aniruddha Patil
>
> On Apr 28, 2020, at 10:05 PM, DImuthu Upeksha 
> wrote:
>
> Thanks. I did the first pass of you PR. Please have a look. Once you
> address the issues, I will do more detailed reviews.
>
> Dimuthu
>
> On Tue, Apr 28, 2020 at 6:22 PM Bafna, Nikita  wrote:
>
>> Hello Dimuthu,
>>
>>
>> Sorry I forgot to mention the PR [1] link. Here it is.
>>
>>
>> [1] https://github.com/apache/airavata-mft/pull/11
>>
>>
>> Regards,
>>
>> Nikita
>>
>>
>> *From: *Bafna, Nikita 
>> *Sent: *Tuesday, April 28, 2020 4:39 PM
>> *To: *dev@airavata.apache.org
>> *Subject: *Re: [External] Re: Airavata MFT- Gdrive transport
>> implementation
>>
>>
>> Hello Dimuthu,
>>
>> We have implemented transport system for Google drive.
>>
>> We have build our transport in a way where if a file with that name
>> already exists it will update that file instead of creating a new one on
>> Google drive.
>>
>> For now we have tested our implementation between endpoints below.
>>
>> S3 to Google drive
>>
>> Google drive to S3
>>
>> Local to Google drive (Screenshot attached)
>>
>> Google drive to Local (Screenshot attached)​
>>
>> Could you please have a look and let us know your feedback.
>>
>> Thank you,
>> Team NSA
>> (Nikita Bafna, Shivali Jejurkar, Aniruddha Patil)
>>
>>
>> --
>> *From:* Bafna, Nikita
>> *Sent:* Tuesday, April 28, 2020 5:20 AM
>> *To:* dev@airavata.apache.org
>> *Subject:* Re: [External] Re: Airavata MFT- Gdrive transport
>> implementation
>>
>>
>> Hi Dimuthu,
>>
>>
>> During testing of the GDrive transport, we came across a situation where
>> if a file is being uploaded on the Gdrive and if the file with same name
>> already exists, then Gdrive creates a new file with the same name (it
>> creates a new version ). So how should this be handled? Should it create a
>> new file or update the existing file?
>>
>>
>> Let us know your inputs.
>>
>>
>> Regards,
>>
>> Nikita
>> --
>> *From:* Bafna, Nikita 
>> *Sent:* Tuesday, April 28, 2020 12:48 AM
>> *To:* dev@airavata.apache.org
>> *Subject:* RE: [External] Re: Airavata MFT- Gdrive transport
>> implementation
>>
>>
>> Okay.
>>
>> Thanks for the inputs Dimuthu.
>>
>>
>> *From:* DImuthu Upeksha 
>> *Sent:* Monday, April 27, 2020 3:05 AM
>> *To:* Airavata Dev 
>> *Subject:* Re: [External] Re: Airavata MFT- Gdrive transport
>> implementation
>>
>>
>> Go with service account approach as this is service to service
>> authentication
>>
>>
>> Dimuthu
>>
>>
>> On Sat, Apr 25, 2020 at 7:11 PM Bafna, Nikita  wrote:
>>
>> Hi Dimuthu,
>>
>>
>> While working for Google drive transport, we are wondering what kind of
>> credential account should we use for our setup.
>>
>>
>> Attaching the options available.
>>
>>
>> Could you please provide your feedback.
>>
>>
>> Regards,
>>
>> Nikita
>> ​
>> --
>>
>> *From:* Suresh Marru 
>> *Sent:* Friday, April 24, 2020 11:36 PM
>> *To:* Airavata Dev
>> *Subject:* [External] Re: Airavata MFT- Gdrive transport implementation
>>
>>
>> This message was sent from a non-IU address. Please exercise caution when
>> clicking links or opening attachments from external sources.
>>
>>
>> A big kudos for all the work on transports, these are nicely evolving.
>>
>>
>> We do not need to drive these with IU decisions, but good point these
>> will lead to potential use cases. On that note, looks like Google Drive
>> will be used for non-sensitive data and one drive for PHI data. So an
>> eventual plan for MFT to support both will be nice. I added it to the
>> spreadsheet.
>>
>>
>> Suresh
>>
>>
>>
>> On Apr 24, 2020, at 12:45 PM, DImuthu Upeksha 
>> wrote:
>>
>>
>> Because of the IUs decision of migration of Box data to Google Drive, I
>> expect the use cases will come into MFT to handle drive transports. So from
>> the usability perspective, I'd prefer gdrive over one drive. But it's up to
>> you to decide. Once you have finalized on one, please update it in [1]
>>
>>
>> [1]
>> https://docs.google.com/spreadsheets/d/1M7-Reda-pCi1l-TSSstI6Yi1pSbtINUqlBFcy5UrOW0/edit#gid=0
>>
>>
>>
>> On Fri, Apr 24, 2020 at 11:43 AM Jejurkar, Shivali 
>> wrote:
>>
>> Hello Dimuthu,
>>
>>
>> I have set up MFT locally and was able to transfer file from S3 to S3 and
>> S3 to local. I am interested in implementing Gdrive transfers. Would MFT
>> also extend support for other cloud storage platforms like OneDrive? Can
>> you share your feedback or any challenges that you see if my team pick
>> Gdrive and Onedrive up?​
>>
>>
>>
>> Thanks,
>>
>> Shivali
>>
>>
>>
>>
>>
>

Re: Passwords and Passphrases

[DImuthu Upeksha]

Hi Gopi,

Thanks for the PR.

I believe this is also related to others who are sending PRs to Airavata
repos. Here are few tips on coding practices in order to contribute to
Airavata

I briefly went through your PR and I can see lots of junk files and
unrelated file changes. I marked few of them and there are lot more in the
PR. This is manly due to committing unrelated eclipse project files and
doing code auto formatting inside the IDE. Different IDEs do different
styles of formatting by default. Of course you can customize them but
personally I would format them by hand whenever possible. It will improve
your coding style. Whichever approach you follow, please have a look at git
diff before committing. If you see unrelated changes like adding lots of
tabs replacing space indentations inside the code, please revert them
before committing. Always try to concentrate your changes regarding your
main purpose of PR. For example. if you saw any bug in core or agent logic,
raise them as separate PRs.

Thanks
Dimuthu

On Tue, Apr 28, 2020 at 11:11 PM GOPI KIRAN  wrote:

> Hi Dimuthu,
>
> I have raised the PR for the FTP implementation.
> Link: https://github.com/apache/airavata-mft/pull/12
>
> With regards,
> Gopi Kiran
>
>
> On Tue, Apr 28, 2020 at 12:27 AM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com> wrote:
>
>> Awesome. Looking forward for your PR
>>
>> On Tue, Apr 28, 2020 at 12:07 AM GOPI KIRAN 
>> wrote:
>>
>>> I agree with you Dimuthu.
>>>
>>> I have written the implementation of FTP storage, resource, and secret
>>> for all types (local DB, file) like it is written for Local and SCP
>>> transport.
>>>
>>> I have finished the FTP implementation and will raise the PR by
>>> tomorrow. If I need to implement the encryption part for the local DB, I
>>> will implement it in the next PR.
>>>
>>> With regards,
>>> Gopi Kiran
>>>
>>>
>>> On Mon, Apr 27, 2020 at 11:47 PM DImuthu Upeksha <
>>> dimuthu.upeks...@gmail.com> wrote:
>>>
>>>> Hi Gopi,
>>>>
>>>> This is implementation specific for secret backend type. If you are
>>>> using the File Backend, you don't need to encrypt as it is the developer
>>>> backend to test the framework so it needs to be simple as possible. But for
>>>> production level backends, yes we should have them encrypted.
>>>>
>>>> Thanks
>>>> Dimuthu
>>>>
>>>> On Mon, Apr 27, 2020 at 10:29 PM GOPI KIRAN 
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Since I need to use a password for FTP and since it's not an SFTP
>>>>> password is the only method of authentication. Should I use Spring 
>>>>> security
>>>>> tools like BCrypt to encrypt the passwords while storing it in local DB or
>>>>> from the file?
>>>>>
>>>>> If that is okay then I would propose to do it with passphrase as well
>>>>> for SCP transfer!
>>>>>
>>>>> With regards,
>>>>> Gopi Kiran
>>>>>
>>>>

Re: [External] Re: Airavata MFT- Gdrive transport implementation

[DImuthu Upeksha]

Thanks. I did the first pass of you PR. Please have a look. Once you
address the issues, I will do more detailed reviews.

Dimuthu

On Tue, Apr 28, 2020 at 6:22 PM Bafna, Nikita  wrote:

> Hello Dimuthu,
>
>
>
> Sorry I forgot to mention the PR [1] link. Here it is.
>
>
>
> [1] https://github.com/apache/airavata-mft/pull/11
>
>
>
> Regards,
>
> Nikita
>
>
>
> *From: *Bafna, Nikita 
> *Sent: *Tuesday, April 28, 2020 4:39 PM
> *To: *dev@airavata.apache.org
> *Subject: *Re: [External] Re: Airavata MFT- Gdrive transport
> implementation
>
>
>
> Hello Dimuthu,
>
> We have implemented transport system for Google drive.
>
> We have build our transport in a way where if a file with that name
> already exists it will update that file instead of creating a new one on
> Google drive.
>
> For now we have tested our implementation between endpoints below.
>
> S3 to Google drive
>
> Google drive to S3
>
> Local to Google drive (Screenshot attached)
>
> Google drive to Local (Screenshot attached)​
>
> Could you please have a look and let us know your feedback.
>
> Thank you,
> Team NSA
> (Nikita Bafna, Shivali Jejurkar, Aniruddha Patil)
>
>
> --
> *From:* Bafna, Nikita
> *Sent:* Tuesday, April 28, 2020 5:20 AM
> *To:* dev@airavata.apache.org
> *Subject:* Re: [External] Re: Airavata MFT- Gdrive transport
> implementation
>
>
> Hi Dimuthu,
>
>
> During testing of the GDrive transport, we came across a situation where
> if a file is being uploaded on the Gdrive and if the file with same name
> already exists, then Gdrive creates a new file with the same name (it
> creates a new version ). So how should this be handled? Should it create a
> new file or update the existing file?
>
>
> Let us know your inputs.
>
>
> Regards,
>
> Nikita
> --
> *From:* Bafna, Nikita 
> *Sent:* Tuesday, April 28, 2020 12:48 AM
> *To:* dev@airavata.apache.org
> *Subject:* RE: [External] Re: Airavata MFT- Gdrive transport
> implementation
>
>
> Okay.
>
> Thanks for the inputs Dimuthu.
>
>
>
> *From:* DImuthu Upeksha 
> *Sent:* Monday, April 27, 2020 3:05 AM
> *To:* Airavata Dev 
> *Subject:* Re: [External] Re: Airavata MFT- Gdrive transport
> implementation
>
>
>
> Go with service account approach as this is service to service
> authentication
>
>
>
> Dimuthu
>
>
>
> On Sat, Apr 25, 2020 at 7:11 PM Bafna, Nikita  wrote:
>
> Hi Dimuthu,
>
>
>
> While working for Google drive transport, we are wondering what kind of
> credential account should we use for our setup.
>
>
>
> Attaching the options available.
>
>
>
> Could you please provide your feedback.
>
>
>
> Regards,
>
> Nikita
>
> ​
> --
>
> *From:* Suresh Marru 
> *Sent:* Friday, April 24, 2020 11:36 PM
> *To:* Airavata Dev
> *Subject:* [External] Re: Airavata MFT- Gdrive transport implementation
>
>
>
> This message was sent from a non-IU address. Please exercise caution when
> clicking links or opening attachments from external sources.
>
>
>
> A big kudos for all the work on transports, these are nicely evolving.
>
>
>
> We do not need to drive these with IU decisions, but good point these will
> lead to potential use cases. On that note, looks like Google Drive will be
> used for non-sensitive data and one drive for PHI data. So an eventual plan
> for MFT to support both will be nice. I added it to the spreadsheet.
>
>
>
> Suresh
>
>
>
> On Apr 24, 2020, at 12:45 PM, DImuthu Upeksha 
> wrote:
>
>
>
> Because of the IUs decision of migration of Box data to Google Drive, I
> expect the use cases will come into MFT to handle drive transports. So from
> the usability perspective, I'd prefer gdrive over one drive. But it's up to
> you to decide. Once you have finalized on one, please update it in [1]
>
>
>
> [1]
> https://docs.google.com/spreadsheets/d/1M7-Reda-pCi1l-TSSstI6Yi1pSbtINUqlBFcy5UrOW0/edit#gid=0
>
>
>
>
> On Fri, Apr 24, 2020 at 11:43 AM Jejurkar, Shivali 
> wrote:
>
> Hello Dimuthu,
>
>
>
> I have set up MFT locally and was able to transfer file from S3 to S3 and
> S3 to local. I am interested in implementing Gdrive transfers. Would MFT
> also extend support for other cloud storage platforms like OneDrive? Can
> you share your feedback or any challenges that you see if my team pick
> Gdrive and Onedrive up?​
>
>
>
>
>
> Thanks,
>
> Shivali
>
>
>
>
>
>

Re: Passwords and Passphrases

[DImuthu Upeksha]

Awesome. Looking forward for your PR

On Tue, Apr 28, 2020 at 12:07 AM GOPI KIRAN  wrote:

> I agree with you Dimuthu.
>
> I have written the implementation of FTP storage, resource, and secret for
> all types (local DB, file) like it is written for Local and SCP transport.
>
> I have finished the FTP implementation and will raise the PR by tomorrow.
> If I need to implement the encryption part for the local DB, I will
> implement it in the next PR.
>
> With regards,
> Gopi Kiran
>
>
> On Mon, Apr 27, 2020 at 11:47 PM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com> wrote:
>
>> Hi Gopi,
>>
>> This is implementation specific for secret backend type. If you are using
>> the File Backend, you don't need to encrypt as it is the developer backend
>> to test the framework so it needs to be simple as possible. But for
>> production level backends, yes we should have them encrypted.
>>
>> Thanks
>> Dimuthu
>>
>> On Mon, Apr 27, 2020 at 10:29 PM GOPI KIRAN 
>> wrote:
>>
>>> Hi,
>>>
>>> Since I need to use a password for FTP and since it's not an SFTP
>>> password is the only method of authentication. Should I use Spring security
>>> tools like BCrypt to encrypt the passwords while storing it in local DB or
>>> from the file?
>>>
>>> If that is okay then I would propose to do it with passphrase as well
>>> for SCP transfer!
>>>
>>> With regards,
>>> Gopi Kiran
>>>
>>

Re: [External] Apache Airavata MFT - OneDrive transport implementation

[DImuthu Upeksha]

Can you summarize the suggested steps to register an onedrive resource in
MFT including registering it in ms graph and fetching tokens? I need to
have an idea on the convenience aspect on this. About the API clients, you
should try to stick to standard clients provided by original providers as
much as possible. If not try to go for a reputed client implementation form
third party. In either case, licenses of these clients should be compatible
with Apache License 2.0. I just had a brief look at ondrive api and it
seems like they don't have a standard java client [2]. However they have a
standard HTTP API [3] so if you use any standard http client, you should be
fine. I saw some people have already done that [4]

[2] https://developer.microsoft.com/en-us/onedrive
[3]
https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_createuploadsession?view=odsp-graph-online
[4] https://www.example-code.com/java/onedrive_download_to_file.asp

On Mon, Apr 27, 2020 at 11:02 PM Patil, Aniruddha Vivek 
wrote:

> Hello Dimuthu,
>
> As best as I understand, the ability of an application to access OneDrive
> depends on whether the application has been registered for MS Graph. (this
> is to avoid a user-dependent login)
> Also, the repository you linked to is a third-party client. Is that
> permissible?
>
> Thanks and regards,
> Aniruddha Patil
>
> On Apr 27, 2020, at 10:39 PM, DImuthu Upeksha 
> wrote:
>
> Hi, Vivek,
>
> I'm little confused about the relation between graph auth and one drive.
> Can't we use a client id/ secret like this [1]?
>
> [1] https://github.com/isac322/OneDrive-SDK-java#1-construct-client-object
>
> Thanks
> Dimuthu
>
> On Mon, Apr 27, 2020 at 5:48 PM Patil, Aniruddha Vivek 
> wrote:
>
>>
>> On further searching, I also came across this, with regard to the 2nd
>> question in the previous mail on this thread. (
>> https://github.com/microsoftgraph/msgraph-sdk-java-core/tree/master)
>>
>> Regards
>> Aniruddha Patil
>>
>> On Apr 27, 2020, at 3:42 PM, Patil, Aniruddha Vivek 
>> wrote:
>>
>> This message was sent from a non-IU address. Please exercise caution when
>> clicking links or opening attachments from external sources.
>>
>> Hello,
>>
>> We’re working on OneDrive transport for Airavata’s MFT and we have the
>> following questions:
>>
>> 1) Whether we should go ahead with this method for authentication? (
>> https://docs.microsoft.com/en-us/graph/auth-v2-service)
>> 2) Whether we should interact with Microsoft Graph with either the REST
>> Api they have provided (
>> https://docs.microsoft.com/en-us/graph/api/overview?view=graph-rest-1.0)
>> or via msgraph-sdk-java? (
>> https://github.com/microsoftgraph/msgraph-sdk-java) (which only has a
>> dev branch and no master, possibly indicating that it hasn’t had a release
>> yet)
>>
>> Thanks and regards,
>> Aniruddha Patil
>>
>>
>>
>

Re: Passwords and Passphrases

[DImuthu Upeksha]

Hi Gopi,

This is implementation specific for secret backend type. If you are using
the File Backend, you don't need to encrypt as it is the developer backend
to test the framework so it needs to be simple as possible. But for
production level backends, yes we should have them encrypted.

Thanks
Dimuthu

On Mon, Apr 27, 2020 at 10:29 PM GOPI KIRAN  wrote:

> Hi,
>
> Since I need to use a password for FTP and since it's not an SFTP
> password is the only method of authentication. Should I use Spring security
> tools like BCrypt to encrypt the passwords while storing it in local DB or
> from the file?
>
> If that is okay then I would propose to do it with passphrase as well for
> SCP transfer!
>
> With regards,
> Gopi Kiran
>

Re: [External] Apache Airavata MFT - OneDrive transport implementation

[DImuthu Upeksha]

Hi, Vivek,

I'm little confused about the relation between graph auth and one drive.
Can't we use a client id/ secret like this [1]?

[1] https://github.com/isac322/OneDrive-SDK-java#1-construct-client-object

Thanks
Dimuthu

On Mon, Apr 27, 2020 at 5:48 PM Patil, Aniruddha Vivek 
wrote:

>
> On further searching, I also came across this, with regard to the 2nd
> question in the previous mail on this thread. (
> https://github.com/microsoftgraph/msgraph-sdk-java-core/tree/master)
>
> Regards
> Aniruddha Patil
>
> On Apr 27, 2020, at 3:42 PM, Patil, Aniruddha Vivek 
> wrote:
>
> This message was sent from a non-IU address. Please exercise caution when
> clicking links or opening attachments from external sources.
>
> Hello,
>
> We’re working on OneDrive transport for Airavata’s MFT and we have the
> following questions:
>
> 1) Whether we should go ahead with this method for authentication? (
> https://docs.microsoft.com/en-us/graph/auth-v2-service)
> 2) Whether we should interact with Microsoft Graph with either the REST
> Api they have provided (
> https://docs.microsoft.com/en-us/graph/api/overview?view=graph-rest-1.0)
> or via msgraph-sdk-java? (
> https://github.com/microsoftgraph/msgraph-sdk-java) (which only has a dev
> branch and no master, possibly indicating that it hasn’t had a release yet)
>
> Thanks and regards,
> Aniruddha Patil
>
>
>

Re: [External] Re: Airavata MFT- Gdrive transport implementation

[DImuthu Upeksha]

Go with service account approach as this is service to service
authentication

Dimuthu

On Sat, Apr 25, 2020 at 7:11 PM Bafna, Nikita  wrote:

> Hi Dimuthu,
>
>
> While working for Google drive transport, we are wondering what kind of
> credential account should we use for our setup.
>
>
> Attaching the options available.
>
>
> Could you please provide your feedback.
>
>
> Regards,
>
> Nikita
>
> ​
> --
> *From:* Suresh Marru 
> *Sent:* Friday, April 24, 2020 11:36 PM
> *To:* Airavata Dev
> *Subject:* [External] Re: Airavata MFT- Gdrive transport implementation
>
> This message was sent from a non-IU address. Please exercise caution when
> clicking links or opening attachments from external sources.
>
> A big kudos for all the work on transports, these are nicely evolving.
>
> We do not need to drive these with IU decisions, but good point these will
> lead to potential use cases. On that note, looks like Google Drive will be
> used for non-sensitive data and one drive for PHI data. So an eventual plan
> for MFT to support both will be nice. I added it to the spreadsheet.
>
> Suresh
>
> On Apr 24, 2020, at 12:45 PM, DImuthu Upeksha 
> wrote:
>
> Because of the IUs decision of migration of Box data to Google Drive, I
> expect the use cases will come into MFT to handle drive transports. So from
> the usability perspective, I'd prefer gdrive over one drive. But it's up to
> you to decide. Once you have finalized on one, please update it in [1]
>
> [1]
> https://docs.google.com/spreadsheets/d/1M7-Reda-pCi1l-TSSstI6Yi1pSbtINUqlBFcy5UrOW0/edit#gid=0
>
>
> On Fri, Apr 24, 2020 at 11:43 AM Jejurkar, Shivali 
> wrote:
>
>> Hello Dimuthu,
>>
>> I have set up MFT locally and was able to transfer file from S3 to S3 and
>> S3 to local. I am interested in implementing Gdrive transfers. Would MFT
>> also extend support for other cloud storage platforms like OneDrive? Can
>> you share your feedback or any challenges that you see if my team pick
>> Gdrive and Onedrive up?​
>>
>>
>> Thanks,
>> Shivali
>>
>>
>>
>

Re: Airavata MFT- Gdrive transport implementation

[DImuthu Upeksha]

Because of the IUs decision of migration of Box data to Google Drive, I
expect the use cases will come into MFT to handle drive transports. So from
the usability perspective, I'd prefer gdrive over one drive. But it's up to
you to decide. Once you have finalized on one, please update it in [1]

[1]
https://docs.google.com/spreadsheets/d/1M7-Reda-pCi1l-TSSstI6Yi1pSbtINUqlBFcy5UrOW0/edit#gid=0


On Fri, Apr 24, 2020 at 11:43 AM Jejurkar, Shivali  wrote:

> Hello Dimuthu,
>
> I have set up MFT locally and was able to transfer file from S3 to S3 and
> S3 to local. I am interested in implementing Gdrive transfers. Would MFT
> also extend support for other cloud storage platforms like OneDrive? Can
> you share your feedback or any challenges that you see if my team pick
> Gdrive and Onedrive up?​
>
>
> Thanks,
> Shivali
>
>
>

Re: [External] Re: Apache Airavata MFT : File Autotransfer

[DImuthu Upeksha]

Only prerequisite is understanding message paths between API, Controller
and Agents. For technology pieces like consul and grpc messaging, you can
easily figure out as there are enough code examples in the code itself.
Once you grab the architecture properly, implementation might be easy or
rather straightforward. I account most of the estimated time for
understanding the core of MFT rather than the implementation part. For
implementation I expect following top level milestones

1. Defining a new transfer type for API [2]
2. Parsing the new transfer type in controller [3]
3. Implementing data structures and timers to handle scheduled transfers in
the controller. This includes handling fault tolerance in case of
controller crashes. This is the place I need a proper design form you on
how you are going to do it

[2]
https://github.com/apache/airavata-mft/blob/master/api/stub/src/main/proto/MFTApi.proto#L9
[3]
https://github.com/apache/airavata-mft/blob/master/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java#L95

Thanks
Dimuthu


On Thu, Apr 23, 2020 at 8:12 PM Patil, Aniruddha Vivek 
wrote:

> Since you mentioned that the project involves complexity, we’d like to know
> 1) If you have a plan in mind that would help us estimate if we could be
> able to realistically complete the project in time.
> 2) A list of prerequisite concepts relevant to this project.
> 3) Our biggest concern is the time constraint due to the end-semester time
> period and hence we wanted to be sure before we commit.
>
> Regards,
> Aniruddha
>
> On Apr 23, 2020, at 5:20 PM, DImuthu Upeksha 
> wrote:
>
> I would say this project is an interesting one to work on if you have some
> quality time to put on.  And I can provide some significant amount of help
> if the time is the critical factor. But I guess narrowing down the scope
> might not work for both the project and you as it would be hard for a new
> one to continue from the point where you have stopped given the complexity
> of the project. It's up to you to decide.
>
> Dimuthu
>
> On Thu, Apr 23, 2020 at 4:20 PM Bafna, Nikita  wrote:
>
>> Hi Dimuthu,
>>
>>
>> Given the short deadline, do you think we can pick up and do small part
>> of out proposal.
>>
>> Or is it okay to take up Google Drive implementation as a transport type?
>>
>>
>> Regards,
>>
>> Nikita
>>
>>
>> *From: *Patil, Aniruddha Vivek 
>> *Sent: *Thursday, April 23, 2020 2:59 PM
>> *To: *dev@airavata.apache.org
>> *Subject: *Re: [External] Re: Apache Airavata MFT : File Autotransfer
>>
>> Hello,
>>
>> As part of our course, we have a soft deadline of 30th April, 2020. We
>> can adjust the scope of this proposal to fit the deadline. However, we
>> would like to use a few more days in order to incorporate this feature
>> completely.
>>
>> Regards,
>> Aniruddha
>>
>> On Apr 23, 2020, at 2:40 PM, DImuthu Upeksha 
>> wrote:
>>
>> Hi,
>>
>> This looks interesting. This is a feature going into the controller of
>> MFT. And you have to define a new Transfer request type for scheduled
>> transfers. Basically it will affect to MFT Controller and MFT API modules.
>> You should have a deep understanding on how messages are processed in
>> controller so I suggest you to setup MFT locally and try out
>> different transfers first. Then you can send a design diagram with some
>> text on how you are going to do it and the timeline for that. Do you have
>> an absolute deadline for this project?
>>
>> [1]
>> https://github.com/apache/airavata-mft/blob/master/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
>>
>> Thanks
>> Dimuthu
>>
>> On Thu, Apr 23, 2020 at 2:24 PM Patil, Aniruddha Vivek 
>> wrote:
>>
>>> Hello Dimuthu,
>>>
>>> Here is a link to our proposal (
>>> https://github.com/airavata-courses/NSA/wiki/Assignment-3-Proposal)
>>> It has more detailed information about this feature.
>>>
>>> Regards,
>>> Aniruddha
>>>
>>> On Apr 22, 2020, at 10:42 PM, DImuthu Upeksha <
>>> dimuthu.upeks...@gmail.com> wrote:
>>>
>>> This message was sent from a non-IU address. Please exercise caution
>>> when clicking links or opening attachments from external sources.
>>>
>>> Hi Aniruddha,
>>>
>>> Can you note down more fine grained goals of this feature. May be add
>>> some use cases as well. Then I will be able to help in figuring out where
>>> to implement it.
>>>
>>> Thanks
>>> Dimuthu
>>>
>>> On Wed, Apr 22, 2020 at 9:37 PM Patil, Aniruddha Vivek 
>>> wrote:
>>>
>>>> Hello,
>>>> As per discussions with Prof. Suresh, we are planning on contributing
>>>> the feature of scheduling file transfers to Airavata's MFT (
>>>> https://www.mail-archive.com/dev@airavata.apache.org/msg12564.html).
>>>> We were considering delivering certain API endpoints that could easily
>>>> be integrated into the dashboard in the future.
>>>>
>>>> Are there any pointers we could follow that would help us with this?
>>>>
>>>> Regards,
>>>> Aniruddha Patil
>>>
>>>
>>>
>>
>

Re: [External] Re: MFT- Dropbox Transport Implementation

[DImuthu Upeksha]

I would say support for both. Do a small validation in the transport. If
user has provided /, use it as it is and if he didn't provide, add a / to
the beginning.

Dimuthu

On Thu, Apr 23, 2020 at 11:46 PM Rajvanshi, Akshay  wrote:

> Hello,
>
>
>
> We had a doubt regarding Dropbox transport implementation. In the
> resources.json file, the Dropbox API requires the resource path to be
> appended with a “/” in the starting as a part of the Dropbox file system
> convention.
>
>
>
> We wanted to ask if we should add the “/” manually or expect the user to
> add this on their own? Also, as a test, we added a “/” on our own while
> parsing the resourcePath, but if the user adds a “/” on their own in the
> resources.json, you get the same error again.
>
>
>
> [image: A screenshot of a computer Description automatically generated]
>
>
>
> Kind Regards
>
> Akshay Rajvanshi
>
>
>
> *From: *DImuthu Upeksha 
> *Reply-To: *"dev@airavata.apache.org" 
> *Date: *Thursday, April 23, 2020 at 14:47
> *To: *Airavata Dev 
> *Subject: *[External] Re: MFT- Dropbox Transport Implementation
>
>
>
> This message was sent from a non-IU address. Please exercise caution when
> clicking links or opening attachments from external sources.
>
>
>
> If you set the md5 to null, it will not be evaluated. So you should be
> fine
>
>
>
> Dimuthu
>
>
>
> On Thu, Apr 23, 2020 at 2:41 PM Ravichandran, Sharanya 
> wrote:
>
> Hi,
>
>
>
> We were taking a look at dropbox's documentation and dropbox's API doesnt
> provide the md5checksum.
>
>
>
> Since we have md5 integrity checks whenever a transfer takes place, can we
> skip the md5 check instead by setting the metadata to null ?
>
>
>
> Thanks,
>
> Sharanya R.
>
>

Re: MFT enhancements

[DImuthu Upeksha]

Hi Gopi,

I guess you got confused over FTP and SFTP [1]. In FTP you can't use ssh
credential to authorize. I believe username and password pair is the only
available authentication mechanism for FTP. I might be wrong. Please
correct me if I do.

[1] https://www.guru99.com/ftp-vs-sftp.html

Thanks
Dimuthu

On Thu, Apr 23, 2020 at 6:44 PM GOPI KIRAN  wrote:

> Shall I take the id_rsa key for logging in to the server, as it is done in
> SCP protocol? This means that the FTP server should be aware of the calling
> system public key!
>
> With regards,
> Gopi Kiran
>
>
> On Thu, Apr 23, 2020 at 6:31 PM GOPI KIRAN  wrote:
>
>> Hi,
>>
>> How do I take the credentials for FTP server? I shouldn't expose the
>> password. ApacheCommons FTP client has only a login method (username &
>> password). Any suggestions?
>>
>> With regards,
>> Gopi Kiran
>>
>>
>> On Thu, Apr 23, 2020 at 2:56 PM DImuthu Upeksha <
>> dimuthu.upeks...@gmail.com> wrote:
>>
>>> If you are interested, mark it in
>>> https://docs.google.com/spreadsheets/d/1M7-Reda-pCi1l-TSSstI6Yi1pSbtINUqlBFcy5UrOW0/edit?usp=sharing
>>>
>>> Dimuthu
>>>
>>> On Thu, Apr 23, 2020 at 2:48 PM GOPI KIRAN 
>>> wrote:
>>>
>>>> Absolutely!
>>>>
>>>> With regards,
>>>> Gopi Kiran
>>>>
>>>>
>>>> On Thu, Apr 23, 2020 at 2:46 PM DImuthu Upeksha <
>>>> dimuthu.upeks...@gmail.com> wrote:
>>>>
>>>>> Hi Gopi,
>>>>>
>>>>> Thanks for the interest. Globus and TUS are little complicated for a
>>>>> new developer for MFT to try out as there are some complicated message
>>>>> paths in both transports. I believe you can start with FTP and go through
>>>>> the pull request process to understand basics of implementing a transport
>>>>> for MFT. Then let's see what we can do on other transports. Will that work
>>>>> for you?
>>>>>
>>>>> Thanks
>>>>> Dimuthu
>>>>>
>>>>> On Thu, Apr 23, 2020 at 2:25 PM GOPI KIRAN 
>>>>> wrote:
>>>>>
>>>>>> Hi Dimuthu,
>>>>>>
>>>>>> I was going over that spreadsheet. Can we go ahead and do our POC on
>>>>>> FTP/Globus/TUS.
>>>>>> It is also said that for SFTP SCP can be reused.
>>>>>>
>>>>>> If those components are still available we will be going ahead with
>>>>>> the POC.
>>>>>> Please confirm.
>>>>>> With regards,
>>>>>> Gopi Kiran
>>>>>>
>>>>>

Re: [External] Re: Apache Airavata MFT : File Autotransfer

[DImuthu Upeksha]

I would say this project is an interesting one to work on if you have some
quality time to put on.  And I can provide some significant amount of help
if the time is the critical factor. But I guess narrowing down the scope
might not work for both the project and you as it would be hard for a new
one to continue from the point where you have stopped given the complexity
of the project. It's up to you to decide.

Dimuthu

On Thu, Apr 23, 2020 at 4:20 PM Bafna, Nikita  wrote:

> Hi Dimuthu,
>
>
>
> Given the short deadline, do you think we can pick up and do small part of
> out proposal.
>
> Or is it okay to take up Google Drive implementation as a transport type?
>
>
>
> Regards,
>
> Nikita
>
>
>
> *From: *Patil, Aniruddha Vivek 
> *Sent: *Thursday, April 23, 2020 2:59 PM
> *To: *dev@airavata.apache.org
> *Subject: *Re: [External] Re: Apache Airavata MFT : File Autotransfer
>
>
> Hello,
>
> As part of our course, we have a soft deadline of 30th April, 2020. We can
> adjust the scope of this proposal to fit the deadline. However, we would
> like to use a few more days in order to incorporate this feature completely.
>
> Regards,
> Aniruddha
>
> On Apr 23, 2020, at 2:40 PM, DImuthu Upeksha 
> wrote:
>
> Hi,
>
> This looks interesting. This is a feature going into the controller of
> MFT. And you have to define a new Transfer request type for scheduled
> transfers. Basically it will affect to MFT Controller and MFT API modules.
> You should have a deep understanding on how messages are processed in
> controller so I suggest you to setup MFT locally and try out
> different transfers first. Then you can send a design diagram with some
> text on how you are going to do it and the timeline for that. Do you have
> an absolute deadline for this project?
>
> [1]
> https://github.com/apache/airavata-mft/blob/master/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
>
> Thanks
> Dimuthu
>
> On Thu, Apr 23, 2020 at 2:24 PM Patil, Aniruddha Vivek 
> wrote:
>
>> Hello Dimuthu,
>>
>> Here is a link to our proposal (
>> https://github.com/airavata-courses/NSA/wiki/Assignment-3-Proposal)
>> It has more detailed information about this feature.
>>
>> Regards,
>> Aniruddha
>>
>> On Apr 22, 2020, at 10:42 PM, DImuthu Upeksha 
>> wrote:
>>
>> This message was sent from a non-IU address. Please exercise caution when
>> clicking links or opening attachments from external sources.
>>
>> Hi Aniruddha,
>>
>> Can you note down more fine grained goals of this feature. May be add
>> some use cases as well. Then I will be able to help in figuring out where
>> to implement it.
>>
>> Thanks
>> Dimuthu
>>
>> On Wed, Apr 22, 2020 at 9:37 PM Patil, Aniruddha Vivek 
>> wrote:
>>
>>> Hello,
>>> As per discussions with Prof. Suresh, we are planning on contributing
>>> the feature of scheduling file transfers to Airavata's MFT (
>>> https://www.mail-archive.com/dev@airavata.apache.org/msg12564.html).
>>> We were considering delivering certain API endpoints that could easily
>>> be integrated into the dashboard in the future.
>>>
>>> Are there any pointers we could follow that would help us with this?
>>>
>>> Regards,
>>> Aniruddha Patil
>>
>>
>>
>

Re: MFT enhancements

[DImuthu Upeksha]

If you are interested, mark it in
https://docs.google.com/spreadsheets/d/1M7-Reda-pCi1l-TSSstI6Yi1pSbtINUqlBFcy5UrOW0/edit?usp=sharing

Dimuthu

On Thu, Apr 23, 2020 at 2:48 PM GOPI KIRAN  wrote:

> Absolutely!
>
> With regards,
> Gopi Kiran
>
>
> On Thu, Apr 23, 2020 at 2:46 PM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com> wrote:
>
>> Hi Gopi,
>>
>> Thanks for the interest. Globus and TUS are little complicated for a new
>> developer for MFT to try out as there are some complicated message paths in
>> both transports. I believe you can start with FTP and go through the pull
>> request process to understand basics of implementing a transport for MFT.
>> Then let's see what we can do on other transports. Will that work for you?
>>
>> Thanks
>> Dimuthu
>>
>> On Thu, Apr 23, 2020 at 2:25 PM GOPI KIRAN 
>> wrote:
>>
>>> Hi Dimuthu,
>>>
>>> I was going over that spreadsheet. Can we go ahead and do our POC on
>>> FTP/Globus/TUS.
>>> It is also said that for SFTP SCP can be reused.
>>>
>>> If those components are still available we will be going ahead with the
>>> POC.
>>> Please confirm.
>>> With regards,
>>> Gopi Kiran
>>>
>>

Re: MFT- Dropbox Transport Implementation

[DImuthu Upeksha]

If you set the md5 to null, it will not be evaluated. So you should be fine

Dimuthu

On Thu, Apr 23, 2020 at 2:41 PM Ravichandran, Sharanya 
wrote:

> Hi,
>
>
> We were taking a look at dropbox's documentation and dropbox's API doesnt
> provide the md5checksum.
>
>
> Since we have md5 integrity checks whenever a transfer takes place, can we
> skip the md5 check instead by setting the metadata to null ?
>
>
> Thanks,
>
> Sharanya R.
>

Re: MFT enhancements

[DImuthu Upeksha]

Hi Gopi,

Thanks for the interest. Globus and TUS are little complicated for a new
developer for MFT to try out as there are some complicated message paths in
both transports. I believe you can start with FTP and go through the pull
request process to understand basics of implementing a transport for MFT.
Then let's see what we can do on other transports. Will that work for you?

Thanks
Dimuthu

On Thu, Apr 23, 2020 at 2:25 PM GOPI KIRAN  wrote:

> Hi Dimuthu,
>
> I was going over that spreadsheet. Can we go ahead and do our POC on
> FTP/Globus/TUS.
> It is also said that for SFTP SCP can be reused.
>
> If those components are still available we will be going ahead with the
> POC.
> Please confirm.
> With regards,
> Gopi Kiran
>

Re: [External] Re: Apache Airavata MFT : File Autotransfer

[DImuthu Upeksha]

Hi,

This looks interesting. This is a feature going into the controller of MFT.
And you have to define a new Transfer request type for scheduled transfers.
Basically it will affect to MFT Controller and MFT API modules. You should
have a deep understanding on how messages are processed in controller so I
suggest you to setup MFT locally and try out different transfers first.
Then you can send a design diagram with some text on how you are going to
do it and the timeline for that. Do you have an absolute deadline for this
project?

[1]
https://github.com/apache/airavata-mft/blob/master/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java

Thanks
Dimuthu

On Thu, Apr 23, 2020 at 2:24 PM Patil, Aniruddha Vivek 
wrote:

> Hello Dimuthu,
>
> Here is a link to our proposal (
> https://github.com/airavata-courses/NSA/wiki/Assignment-3-Proposal)
> It has more detailed information about this feature.
>
> Regards,
> Aniruddha
>
> On Apr 22, 2020, at 10:42 PM, DImuthu Upeksha 
> wrote:
>
> This message was sent from a non-IU address. Please exercise caution when
> clicking links or opening attachments from external sources.
>
> Hi Aniruddha,
>
> Can you note down more fine grained goals of this feature. May be add some
> use cases as well. Then I will be able to help in figuring out where to
> implement it.
>
> Thanks
> Dimuthu
>
> On Wed, Apr 22, 2020 at 9:37 PM Patil, Aniruddha Vivek 
> wrote:
>
>> Hello,
>> As per discussions with Prof. Suresh, we are planning on contributing the
>> feature of scheduling file transfers to Airavata's MFT (
>> https://www.mail-archive.com/dev@airavata.apache.org/msg12564.html).
>> We were considering delivering certain API endpoints that could easily be
>> integrated into the dashboard in the future.
>>
>> Are there any pointers we could follow that would help us with this?
>>
>> Regards,
>> Aniruddha Patil
>
>
>

Re: Apache Airavata MFT : File Autotransfer

[DImuthu Upeksha]

Hi Aniruddha,

Can you note down more fine grained goals of this feature. May be add some
use cases as well. Then I will be able to help in figuring out where to
implement it.

Thanks
Dimuthu

On Wed, Apr 22, 2020 at 9:37 PM Patil, Aniruddha Vivek 
wrote:

> Hello,
> As per discussions with Prof. Suresh, we are planning on contributing the
> feature of scheduling file transfers to Airavata's MFT (
> https://www.mail-archive.com/dev@airavata.apache.org/msg12564.html).
> We were considering delivering certain API endpoints that could easily be
> integrated into the dashboard in the future.
>
> Are there any pointers we could follow that would help us with this?
>
> Regards,
> Aniruddha Patil

Re: [External] Re: Apache Airavata MFT - AWS/GCS support

[DImuthu Upeksha]

If you look at the error line, it's calling super.getS3Secret(request,
responseObserver); which should not be done in GRPC services. This will not
give client side errors as the client has got all the data it needed in
earlier lines. I will remove this invocation and commit. Thanks for
reporting this

Dimuthu

On Tue, Apr 21, 2020 at 9:11 PM Rajvanshi, Akshay  wrote:

> Hello,
>
>
>
> In addition to the previous thread from Aravind regarding the error, we
> tested the implementation from apache repository directly without making
> any of our own changes and did testing with other protocols and faced the
> similar problem.
>
>
>
> Kind Regards
>
> Akshay Rajvanshi
>
>
>
> *From: *Aravind Ramalingam 
> *Reply-To: *"dev@airavata.apache.org" 
> *Date: *Tuesday, April 21, 2020 at 20:58
> *To: *"dev@airavata.apache.org" 
> *Subject: *Re: [External] Re: Apache Airavata MFT - AWS/GCS support
>
>
>
> Hello,
>
>
>
> While testing we noticed an error in the SecretServiceApplication, it
> seems to be a problem with the gRPC calls to the service.
>
>
>
> I have attached the screenshot for your reference.
>
>
>
> Could you please help us with this?
>
>
>
> Thank you
>
> Aravind Ramalingam
>
>
>
>
>
>
>
> On Mon, Apr 20, 2020 at 10:59 PM Aravind Ramalingam 
> wrote:
>
> Hi Dimuthu,
>
>
>
> Thank you for the review. We will look into the changes asap.
>
>
>
> Thank you
>
> Aravind Ramalingam
>
>
>
> On Apr 20, 2020, at 22:42, DImuthu Upeksha 
> wrote:
>
> Hi Aravind,
>
>
>
> I reviewed the PR and submitted my reviews. Please have a look at them. I
> didn't thoroughly go through optimizations in the code as there are some
> templating fixes and cleaning up required. Once you fix them, I will do a
> thorough review. Make sure to do a rebase of the PR next time as there are
> conflicts from other commits. Thanks for your contributions.
>
>
>
> Dimuthu
>
>
>
> On Sun, Apr 19, 2020 at 10:13 PM Aravind Ramalingam 
> wrote:
>
> Hello,
>
>
>
> We have raised a Pull Request [12].
>
>
>
> We look forward to your feedback.
>
>
>
> [12] https://github.com/apache/airavata-mft/pull/6
>
>
>
> Thank you
>
> Aravind Ramalingam
>
>
>
> On Sun, Apr 19, 2020 at 8:32 PM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com> wrote:
>
> Sounds good. Please send a PR once it is done.
>
>
>
> Dimuthu
>
>
>
> On Sun, Apr 19, 2020 at 7:23 PM Aravind Ramalingam 
> wrote:
>
> Hello,
>
>
>
> Thank you Sudhakar and Dimuthu. We figured it out.
>
>
>
> Like Sudhakar had pointed out with the issue link, GCS had returned a
> BASE64 Md5Hash, we had to convert it to HEX and it matched with the S3 hash.
>
>
>
> Currently we successfully tested from S3 to GCS and back. We are yet to
> test with other protocols.
>
>
>
> Thank you
>
> Aravind Ramalingam
>
>
>
> On Sun, Apr 19, 2020 at 4:58 PM Pamidighantam, Sudhakar 
> wrote:
>
> https://github.com/googleapis/google-cloud-java/issues/4117 Does this
> help?
>
>
>
> Thanks,
>
> Sudhakar.
>
>
>
> *From: *DImuthu Upeksha 
> *Reply-To: *"dev@airavata.apache.org" 
> *Date: *Sunday, April 19, 2020 at 4:46 PM
> *To: *Airavata Dev 
> *Subject: *Re: [External] Re: Apache Airavata MFT - AWS/GCS support
>
>
>
> Aravind,
>
>
>
> Can you send a PR for what you have done so far so that I can provide a
> feedback. One thing you have to make sure is that the GCS Metadata
> collector returns the correct md5 for that file. You can download the file
> and run "md5sum " locally to get actual md5 value for that file
> and compare with what you can see in GCS implementation.
>
>
>
> In S3, etag is the right property to fetch md5 for target resource. I'm
> not sure what is the right method for GCS. You have to locally try and
> verify.
>
>
>
> Thanks
>
> Dimuthu
>
>
>
> On Sun, Apr 19, 2020 at 3:32 PM Aravind Ramalingam 
> wrote:
>
> Hi Dimuthu,
>
>
>
> We are working on GCS and we got certain parts working, but after a
> transfer is compete we are facing errors with the metadata checks.
>
>
>
> 
>
>
>
> We are currently testing S3 to GCS. We noticed in the S3 implementation
> that Etag was set as the Md5sum. In our case we tried using both Etag and
> Md5Hash, but both threw the above error.
>
>
>
> //S3 implementation
>
> metadata.setMd5sum(s3Metadata.getETag());
>
> //GCS implementation
>
> metadata.setMd5sum(gcsMetadata.getEta

Re: Queries on MFT containerization

[DImuthu Upeksha]

Hi Shivam,

I will reply inline

On Sun, Apr 19, 2020 at 12:06 PM Rastogi, Shivam  wrote:

> Hi Team,
>
>
> For our final project, we are trying to containerize and deploy MFT
> services in a service-mesh.
>
>
> Could you please advise on the following.
>
>
> 1. Why did we use Consul in MFT, instead of Kafka or some other message
> broker? Trying to understand the advantage that Consul provides. As per
> our understanding, Consul is only used as a message broker(key value store)
>
>
>
Consul is used for multiple use cases in MFT. Asynchronous messaging is
one. We use it for leader election and distributed locks as well. For
example, framework allows only one agent to connect to the framework with a
unique name using distributed locks. And in future we plan to provide the
leader election among multiple controllers through consul.


> 2. As per our initial understanding, there are 5 services + 1 consul
> service in MFT. Do you think there will be any issues if we segregate the
> MFT code in 5 different docker containers + 1 container for the consul
> service? Service breakdown is given below.
>
>
>
>1. API service - Used by clients for submitting/fetching requests
>   2. Controller - Communicates with Consul for kvs, accepts live
>   agents
>   3. Resource service - Fetching details of the resources(files) that
>   need to be transferred
>   4. Secret service - Storing/Fetching the credentials for different
>   services
>   5. Agent - Manages the transfer of files, connections with external
>   services
>   6. Consul - Used for communication between different services,
>   using a key-value pair system (something similar to Kafka).
>
>
>
Your understanding is correct. However for Agents, identity is critical. So
if you are using kubernetes to deploy them, try to use stateful sets to
deploy agents

[1] https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

Please let us know your views.
>
>
>

MFT Transport Implementation Status

[DImuthu Upeksha]

Hi Folks,

I'm really glad to see some of you have already started working on
implementing transports for MFT. So I thought of sharing a spreadsheet [1]
to track who is doing what to make sure that two persons are not working on
the same transport. Please go ahead and update the document if you are
working on any.

[1]
https://docs.google.com/spreadsheets/d/1M7-Reda-pCi1l-TSSstI6Yi1pSbtINUqlBFcy5UrOW0/edit?usp=sharing

Thanks
Dimuthu

Re: [airavata-mft] Box transport agent for airavata mft

[DImuthu Upeksha]

Hi Dinuka,

I reviewed your pending PR and it looks really good. I'm glad that you have
followed standard coding style to make this more consistent with the rest
of the code. I will reply inline for your comments

[9] https://github.com/apache/airavata-mft/pull/5

Thanks
Dimuthu

On Sun, Apr 19, 2020 at 11:26 AM Apache Airavata 
wrote:

> Hi Dinuka,
>
> Thanks for the nice summary and for your contributions. These style of
> communication is a good example for GSOC students to follow.
>
> Suresh
>
> On Apr 19, 2020, at 10:17 AM, Dinuka Desilva 
> wrote:
>
> 
> Dear community,
>
> I started working on box transport for airavata-mft agent. For now, let me
> share my findings.
>
> There's an SDK available as a maven dependency [2] and they do have a
> comprehensive documentation [3].
>
> So, far I couldn't find a way to create or recreate files by path. Instead
> it supports only recreating an existing file by id [4] or uploading a new
> file to an identified folder which requires both folder id and file id
> [4][5], So, assuming creation by path is not supported, both source and
> destination files have to be available before the transport.
>
> Looking at the documentation, they support us to provide the file id when
uploading the file. Can't we use as the alternative for resource path?


> One limitation found in uploading was that uploading chunks is limited to
> files larger than 20mb [1]. So, the transport method has to check the file
> size and decide whether to uploaded as chunks or not.
>
> Still I'm working on to figure out an authentication method [6]. I could
> find only dev token [8] to be working yet. They provide an app token, which
> seems limited only to some api calls [7]. And the other options are JWT
> and Oauth2. While Oauth2 is out of the scope, JWT has many options to get a
> connection without a login though neither worked to me yet.
>
> References
> [1] https://developer.box.com/guides/uploads/chunked/
> [2] http://opensource.box.com/box-java-sdk/
> [3] https://github.com/box/box-java-sdk/blob/master/doc/files.md
> [4]
> https://github.com/box/box-java-sdk/blob/master/doc/files.md#upload-a-large-file-version-in-chunks
> [5]
> https://github.com/box/box-java-sdk/blob/master/doc/folders.md#get-a-folders-information
> [6] https://github.com/box/box-java-sdk/blob/master/doc/authentication.md
> [7] https://developer.box.com/guides/embed/box-view/
> [8]
> https://github.com/box/box-java-sdk/blob/master/doc/authentication.md#developer-token
>
> Regards,
> Dinuka
>
>

Re: [airavata-mft] Separating mft-agent module to a golang implementation

[DImuthu Upeksha]

Hi Dinuka,

I will reply inline

On Fri, Apr 17, 2020 at 12:37 PM Dinuka Desilva 
wrote:

> Hi,
>
> I'm currently looking in to convert the existing java implementation of
> agent module to golang considering better concurrency, performance and make
> it more platform enabled. Before just starting up, I'm trying to
> discuss which kind of changes has to be done on which end.
>
> 1) I believe we are considering to accommodate multiple agents available
> per controller and each agent to support multiple transports. Which means,
> it's a collection of agents connected to a one controller. I think this is
> not done yet.
>
> There should be only one active controller for entire system. Controller
is the master and agents are the workers. This follows the master slave
architecture. Controller sends messages to agents when a transfer needs to
be performed. And this is already in place.

2) Currently my understanding according to the implementation is that the
> agent module is kind of coupled with the api and controller. I feel that
> agent has to be an independent module. Or am I missing something?
>
>
Agent is not coupled with the controller. But it has a standard messaging
API built on top of Consul. If controller needs to get some work done by
the agent, it has to communicate in that particular messaging API. But
there is no code level dependency between the controller and agents.
However Agent has dependencies with Resource and Secret APIs to fetch
metadata of the resources. This is also not with the Service but with the
APIs.


> 3) The agent being independent is required for the agent to be converted
> to golang too.
>
>
Yes agent has to be independent from the code level with other components
of the framework but it should have API level dependencies to communicate
with the rest of the system. These are implemented on top of protobuf for
Resource and Secret services which can be easily built for golang and
consul level messaging dependencies with the controller which can be
fulfilled through a golang consul client or just following consul HTTP API.

4) I was also thinking whether the combination of transports available per
> agent also should be independently installable. So, that each agent could
> decide which transports are required. Plus they could introduce new
> transport plugins. (A kind of an enhancement)
>

+1 and this is a good feature to have in agents. Agent have a feature [1]
to inform controller about supported protocols but this not still validated
from the controller when submitting a transfer job. We need to implement
this feature

[1]
https://github.com/apache/airavata-mft/blob/master/agent/src/main/java/org/apache/airavata/mft/agent/MFTAgent.java#L245


>
> 5) I'm just doubt about the "LOCAL" transport. how does it work with
> multiple agents. According to the architecture where does the local storage
> exist? Is it inside the agent?
>

Local resource is attached to the Agent. If and Agent receives a local
resource transfer request, it looks at the local file system of the machine
where it was installed.


> Regards,
> Dinuka
>

Re: [External] Re: Apache Airavata MFT - AWS/GCS support

[DImuthu Upeksha]

Hi Aravind,

You don't need the file to be present in the gcs example I sent. It needs
an Input Stream to read the content. You can use the same approach I have
done in S3 [9] transport to do that. It's straightforward. Replace file
input stream with context.getStreamBuffer().getInputStream().

Akshay,

You can't assume that file is on the machine. It should be provided from
the secret service. I found this example in [10]

Storage storage = StorageOptions.newBuilder()
.setCredentials(ServiceAccountCredentials.fromStream(new
FileInputStream("/path/to/my/key.json")))
.build()
.getService();


It accepts a InputStream of json. You can programmatically load the content
of that json into a java String through secret service and convert that
string to a Input Stream as shown in [11]

[9]
https://github.com/apache/airavata-mft/blob/master/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3Sender.java#L73
[10] https://github.com/googleapis/google-cloud-java
[11] https://www.baeldung.com/convert-string-to-input-stream

Thanks
Dimuthu

On Sun, Apr 19, 2020 at 2:03 AM Rajvanshi, Akshay  wrote:

> Hello,
>
>
>
> We were searching about how to use google API’s to send files, but it’s
> required the first steps to be authentication. In that, the GCP API
> requires a credentials.json file to be present in the system.
>
>
>
> Is it fine if we currently design the GCS transport feature such that the
> file is already present in the system ?
>
>
>
> Kind Regards
>
> Akshay
>
>
>
> *From: *Aravind Ramalingam 
> *Reply-To: *"dev@airavata.apache.org" 
> *Date: *Friday, April 17, 2020 at 00:30
> *To: *"dev@airavata.apache.org" 
> *Subject: *[External] Re: Apache Airavata MFT - AWS/GCS support
>
>
>
> This message was sent from a non-IU address. Please exercise caution when
> clicking links or opening attachments from external sources.
>
>
> Hello,
>
>
>
> Wouldn't it be that in this example the whole file has to be present and
> converted into a single stream and uploaded at once?
>
> We had understood that MFT expects it to be chunk by chunk upload without
> having to have the entire file present.
>
>
>
> Thank you
>
> Aravind Ramalingam
>
>
>
> On Apr 17, 2020, at 00:07, DImuthu Upeksha 
> wrote:
>
> Aravind,
>
>
>
> Streaming is supported in GCS java client. Have a look at here [8]
>
>
>
> [8]
> https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/storage/json-api/src/main/java/StorageSample.java#L104
>
>
>
> Thanks
>
> Dimuthu
>
>
>
> On Thu, Apr 16, 2020 at 9:56 PM Aravind Ramalingam 
> wrote:
>
> Hello Dimuthu,
>
>
>
> As a followup, we explored GCS in detail. We are faced with a small
> dilemma. We found that though GCS has a Java support, but the functionality
> does not seem to extend to a stream based upload and download.
>
> The documentation says it is currently done with a gsutil command line
> library [7], hence we are confused if we would be able to proceed the GCS
> integration.
>
>
>
> Could you please give us any suggestions? Also we were wondering if we
> could maybe take up Box integration or some other provider if GCS proves
> not possible currently.
>
>
>
> [7] https://cloud.google.com/storage/docs/streaming
>
>
>
> Thank you
>
> Aravind Ramalingam
>
>
>
> On Thu, Apr 16, 2020 at 12:45 AM Aravind Ramalingam 
> wrote:
>
> Hello Dimuthu,
>
>
>
> We had just started looking into Azure and GCS. Since Azure is done we
> will take up and explore GCS.
>
>
>
> Thank you for the update.
>
> Thank you
>
> Aravind Ramalingam
>
>
>
> On Apr 16, 2020, at 00:30, DImuthu Upeksha 
> wrote:
>
> Aravind,
>
>
>
> I'm not sure whether you have made any progress on Azure transport yet. I
> got a chance to look into that [6]. Let me know if you are working on GCS
> or any other so that I can plan ahead. Next I will be focusing on Box
> transport.
>
>
>
> [6]
> https://github.com/apache/airavata-mft/commit/013ed494eb958990d0a6f90186a53103e1237bcd
>
>
>
> Thanks
>
> Dimuthu
>
>
>
> On Mon, Apr 6, 2020 at 5:19 PM Aravind Ramalingam 
> wrote:
>
> Hi  Dimuthu,
>
>
>
> Thank you for the update. We look into it and get an idea about how the
> system works.
>
> We were hoping to try an implementation for GCS, we will also look into
> Azure.
>
>
>
> Thank you
>
> Aravind Ramalingam
>
>
>
> On Mon, Apr 6, 2020 at 4:44 PM DImuthu Upeksha 
> wrote:
>
> Aravind,
>
>
>
> Here [2] is the complete commit for S3 transport imp

Re: Apache Airavata MFT - AWS/GCS support

[DImuthu Upeksha]

Aravind,

Streaming is supported in GCS java client. Have a look at here [8]

[8]
https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/storage/json-api/src/main/java/StorageSample.java#L104

Thanks
Dimuthu

On Thu, Apr 16, 2020 at 9:56 PM Aravind Ramalingam 
wrote:

> Hello Dimuthu,
>
> As a followup, we explored GCS in detail. We are faced with a small
> dilemma. We found that though GCS has a Java support, but the functionality
> does not seem to extend to a stream based upload and download.
> The documentation says it is currently done with a gsutil command line
> library [7], hence we are confused if we would be able to proceed the GCS
> integration.
>
> Could you please give us any suggestions? Also we were wondering if we
> could maybe take up Box integration or some other provider if GCS proves
> not possible currently.
>
> [7] https://cloud.google.com/storage/docs/streaming
>
> Thank you
> Aravind Ramalingam
>
> On Thu, Apr 16, 2020 at 12:45 AM Aravind Ramalingam 
> wrote:
>
>> Hello Dimuthu,
>>
>> We had just started looking into Azure and GCS. Since Azure is done we
>> will take up and explore GCS.
>>
>> Thank you for the update.
>>
>> Thank you
>> Aravind Ramalingam
>>
>> On Apr 16, 2020, at 00:30, DImuthu Upeksha 
>> wrote:
>>
>> 
>> Aravind,
>>
>> I'm not sure whether you have made any progress on Azure transport yet. I
>> got a chance to look into that [6]. Let me know if you are working on GCS
>> or any other so that I can plan ahead. Next I will be focusing on Box
>> transport.
>>
>> [6]
>> https://github.com/apache/airavata-mft/commit/013ed494eb958990d0a6f90186a53103e1237bcd
>>
>> Thanks
>> Dimuthu
>>
>> On Mon, Apr 6, 2020 at 5:19 PM Aravind Ramalingam 
>> wrote:
>>
>>> Hi  Dimuthu,
>>>
>>> Thank you for the update. We look into it and get an idea about how the
>>> system works.
>>> We were hoping to try an implementation for GCS, we will also look into
>>> Azure.
>>>
>>> Thank you
>>> Aravind Ramalingam
>>>
>>> On Mon, Apr 6, 2020 at 4:44 PM DImuthu Upeksha <
>>> dimuthu.upeks...@gmail.com> wrote:
>>>
>>>> Aravind,
>>>>
>>>> Here [2] is the complete commit for S3 transport implementation but
>>>> don't get confused by the amount of changes as this includes both transport
>>>> implementation and the service backend implementations. If you need to
>>>> implement a new transport, you need to implement a Receiver, Sender and a
>>>> MetadataCollector like this [3]. Then you need to add that resource support
>>>> to Resource service and Secret service [4] [5]. You can similarly do that
>>>> for Azure. A sample SCP -> S3 transfer request is like below. Hope that
>>>> helps.
>>>>
>>>> String sourceId = "remote-ssh-resource";
>>>> String sourceToken = "local-ssh-cred";
>>>> String sourceType = "SCP";
>>>> String destId = "s3-file";
>>>> String destToken = "s3-cred";
>>>> String destType = "S3";
>>>>
>>>> TransferApiRequest request = TransferApiRequest.newBuilder()
>>>> .setSourceId(sourceId)
>>>> .setSourceToken(sourceToken)
>>>> .setSourceType(sourceType)
>>>> .setDestinationId(destId)
>>>> .setDestinationToken(destToken)
>>>> .setDestinationType(destType)
>>>> .setAffinityTransfer(false).build();
>>>>
>>>>
>>>> [2]
>>>> https://github.com/apache/airavata-mft/commit/62fae3d0ab2921fa8bf0bea7970e233f842e6948
>>>> [3]
>>>> https://github.com/apache/airavata-mft/tree/master/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3
>>>> [4]
>>>> https://github.com/apache/airavata-mft/blob/master/services/resource-service/stub/src/main/proto/ResourceService.proto#L90
>>>> [5]
>>>> https://github.com/apache/airavata-mft/blob/master/services/secret-service/stub/src/main/proto/SecretService.proto#L45
>>>>
>>>> Thanks
>>>> Dimuthu
>>>>
>>>>
>>>> On Sun, Apr 5, 2020 at 12:10 AM DImuthu Upeksha <
>>>> dimuthu.upeks...@gmail.com> wrote:
>>>>
>>>>> There is a working on S3 transport in my local copy. Will commit it
>>>>> once I te

Re: Apache Airavata MFT - AWS/GCS support

[DImuthu Upeksha]

Aravind,

I'm not sure whether you have made any progress on Azure transport yet. I
got a chance to look into that [6]. Let me know if you are working on GCS
or any other so that I can plan ahead. Next I will be focusing on Box
transport.

[6]
https://github.com/apache/airavata-mft/commit/013ed494eb958990d0a6f90186a53103e1237bcd

Thanks
Dimuthu

On Mon, Apr 6, 2020 at 5:19 PM Aravind Ramalingam  wrote:

> Hi  Dimuthu,
>
> Thank you for the update. We look into it and get an idea about how the
> system works.
> We were hoping to try an implementation for GCS, we will also look into
> Azure.
>
> Thank you
> Aravind Ramalingam
>
> On Mon, Apr 6, 2020 at 4:44 PM DImuthu Upeksha 
> wrote:
>
>> Aravind,
>>
>> Here [2] is the complete commit for S3 transport implementation but don't
>> get confused by the amount of changes as this includes both transport
>> implementation and the service backend implementations. If you need to
>> implement a new transport, you need to implement a Receiver, Sender and a
>> MetadataCollector like this [3]. Then you need to add that resource support
>> to Resource service and Secret service [4] [5]. You can similarly do that
>> for Azure. A sample SCP -> S3 transfer request is like below. Hope that
>> helps.
>>
>> String sourceId = "remote-ssh-resource";
>> String sourceToken = "local-ssh-cred";
>> String sourceType = "SCP";
>> String destId = "s3-file";
>> String destToken = "s3-cred";
>> String destType = "S3";
>>
>> TransferApiRequest request = TransferApiRequest.newBuilder()
>> .setSourceId(sourceId)
>> .setSourceToken(sourceToken)
>> .setSourceType(sourceType)
>> .setDestinationId(destId)
>> .setDestinationToken(destToken)
>> .setDestinationType(destType)
>> .setAffinityTransfer(false).build();
>>
>>
>> [2]
>> https://github.com/apache/airavata-mft/commit/62fae3d0ab2921fa8bf0bea7970e233f842e6948
>> [3]
>> https://github.com/apache/airavata-mft/tree/master/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3
>> [4]
>> https://github.com/apache/airavata-mft/blob/master/services/resource-service/stub/src/main/proto/ResourceService.proto#L90
>> [5]
>> https://github.com/apache/airavata-mft/blob/master/services/secret-service/stub/src/main/proto/SecretService.proto#L45
>>
>> Thanks
>> Dimuthu
>>
>>
>> On Sun, Apr 5, 2020 at 12:10 AM DImuthu Upeksha <
>> dimuthu.upeks...@gmail.com> wrote:
>>
>>> There is a working on S3 transport in my local copy. Will commit it once
>>> I test it out properly. You can follow the same pattern for any cloud
>>> provider which has clients with streaming IO. Streaming among different
>>> transfer protocols inside an Agent has been discussed in the last part of
>>> this [1] document. Try to get the conceptual idea from that and reverse
>>> engineer SCP transport.
>>>
>>> [1]
>>> https://docs.google.com/document/d/1zrO4Z1dn7ENhm1RBdVCw-dDpWiebaZEWy66ceTWoOlo
>>>
>>> Dimuthu
>>>
>>> On Sat, Apr 4, 2020 at 9:22 PM Aravind Ramalingam 
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> We were looking at the existing code in the project. We could find
>>>> implementations only for local copy and SCP.
>>>> We were confused on how to go about with an external provider like S3
>>>> or Azure? Since it would require integrating with their respective clients.
>>>>
>>>> Thank you
>>>> Aravind Ramalingam
>>>>
>>>> > On Apr 4, 2020, at 21:15, Suresh Marru  wrote:
>>>> >
>>>> > Hi Aravind,
>>>> >
>>>> > I have to catch up with the code, but you may want to look at the S3
>>>> implementation and extend it to Azure, GCP or other cloud services like
>>>> Box, Dropbox and so on.
>>>> >
>>>> > There could be many use cases, here is an idea:
>>>> >
>>>> > * Compute a job on a supercomputer with SCP access and push the
>>>> outputs to a Cloud storage.
>>>> >
>>>> > Suresh
>>>> >
>>>> >> On Apr 4, 2020, at 8:09 PM, Aravind Ramalingam 
>>>> wrote:
>>>> >>
>>>> >> Hello,
>>>> >>
>>>> >> We set up the MFT project on local system and tested out SCP
>>>> transfer between JetStream VMs, we were wondering how the support can be
>>>> extended for AWS/GCS.
>>>> >>
>>>> >> As per our understanding, the current implementation has support for
>>>> two protocols i.e. local-transport and scp-transport. Would we have to
>>>> modify/add to the code base to extend support for AWS/GCS clients?
>>>> >>
>>>> >> Could you please provide suggestions for this use case.
>>>> >>
>>>> >> Thank you
>>>> >> Aravind Ramalingam
>>>> >
>>>>
>>>

Re: [airavata-mft] Null pointer exception on LOCAL to LOCAL transport

[DImuthu Upeksha]

Hi Dinuka,

It should be fixed by now. Local transport was not enhanced for the
latest architecture. This [1] commit should fix it

[1]
https://github.com/apache/airavata-mft/commit/380a674d904ceab21a32c353b74c3742472801ec

Thanks
Dimuthu

On Wed, Apr 15, 2020 at 4:32 PM Dinuka Desilva 
wrote:

> Hi,
>
> on airavata-mft, I'm trying to make a transport from local to local and
> I'm getting a null pointer exception. I'm doubt whether there's any
> configuration miss on my end. If not whether the local transport is
> correctly working at the moment. I'm running the latest master branch code.
>
>
> *resources.json*
> [
>   {
> "type": "LOCAL",
> "resourceId":  "abcd-txt-file",
> "resourcePath": "/tmp/abcd.txt"
>   },
>   {
> "type": "LOCAL",
> "resourceId":  "efgh-txt-file",
> "resourcePath": "/tmp/efgh.txt"
>   }
> ]
>
>
>
> *SampleClient.java*
> import org.apache.airavata.mft.api.client.MFTApiClient;
> import org.apache.airavata.mft.api.service.*;
>
> public class SampleClient {
> public static void main(String args[]) throws InterruptedException {
>
> MFTApiServiceGrpc.MFTApiServiceBlockingStub client =
> MFTApiClient.buildClient("localhost", 7004);
>
> String sourceId = "abcd-txt-file";
> String sourceToken = "local-ssh-cred";
> String destId = "efgh-txt-file";
> String destToken = "local-ssh-cred";
>
> TransferApiRequest request = TransferApiRequest.newBuilder()
> .setSourceId(sourceId)
> .setSourceType("LOCAL")
> .setDestinationId(destId)
> .setDestinationType("LOCAL")
> .setAffinityTransfer(false).build();
>
> // Submitting the transfer to MFT
> TransferApiResponse transferApiResponse =
> client.submitTransfer(request);
> while(true) {
> // Monitoring transfer status
> try {
> TransferStateApiResponse transferState =
> client.getTransferState(TransferStateApiRequest.newBuilder().setTransferId(transferApiResponse.getTransferId()).build());
> System.out.println("Latest Transfer State " +
> transferState.getState());
> System.out.println("Latest Transfer State " +
> transferState.getDescription());
>
> } catch (Exception e) {
> System.out.println("Errored " + e.getMessage());
> }
> Thread.sleep(1000);
> }
> }
> }
>
>
> *Log*
>
> 01:53:22.937 [grpc-nio-worker-ELG-1-2] DEBUG
> io.grpc.netty.NettyClientHandler - [id: 0xb581d434, L:/127.0.0.1:61994 -
> R:localhost/127.0.0.1:7004] OUTBOUND HEADERS: streamId=31
> headers=GrpcHttp2OutboundHeaders[:authority: localhost:7004, :path:
> /org.apache.airavata.mft.api.service.MFTApiService/getTransferState,
> :method: POST, :scheme: http, content-type: application/grpc, te: trailers,
> user-agent: grpc-java-netty/1.25.0, grpc-accept-encoding: gzip]
> streamDependency=0 weight=16 exclusive=false padding=0 endStream=false
>
> 01:53:22.938 [grpc-nio-worker-ELG-1-2] DEBUG
> io.grpc.netty.NettyClientHandler - [id: 0xb581d434, L:/127.0.0.1:61994 -
> R:localhost/127.0.0.1:7004] OUTBOUND DATA: streamId=31 padding=0
> endStream=true length=43
> bytes=260a2466303565383630382d363536302d346463332d396365612d34373364383232656362
>
> 01:53:22.942 [grpc-nio-worker-ELG-1-2] DEBUG
> io.grpc.netty.NettyClientHandler - [id: 0xb581d434, L:/127.0.0.1:61994 -
> R:localhost/127.0.0.1:7004] INBOUND HEADERS: streamId=31
> headers=GrpcHttp2ResponseHeaders[:status: 200, content-type:
> application/grpc, grpc-encoding: identity, grpc-accept-encoding: gzip]
> streamDependency=0 weight=16 exclusive=false padding=0 endStream=false
>
> 01:53:22.943 [grpc-nio-worker-ELG-1-2] DEBUG
> io.grpc.netty.NettyClientHandler - [id: 0xb581d434, L:/127.0.0.1:61994 -
> R:localhost/127.0.0.1:7004] INBOUND DATA: streamId=31 padding=0
> endStream=false length=1196
> bytes=0004a70a064641494c45441093918afc972e2295096a6176612e6c616e672e4e756c6c506f696e746572457863657074696f6e0a096174206f72672e6170...
>
> 01:53:22.943 [grpc-nio-worker-ELG-1-2] DEBUG
> io.grpc.netty.NettyClientHandler - [id: 0xb581d434, L:/127.0.0.1:61994 -
> R:localhost/127.0.0.1:7004] INBOUND HEADERS: streamId=31
> headers=GrpcHttp2ResponseHeaders[grpc-status: 0] streamDependency=0
> weight=16 exclusive=false padding=0 endStream=true
>
> Latest Transfer State FAILED
>
> Latest Transfer State java.lang.NullPointerException
>
> at
> org.apache.airavata.mft.transport.local.LocalMetadataCollector.getGetResourceMetadata(LocalMetadataCollector.java:52)
>
> at
> org.apache.airavata.mft.agent.TransportMediator.transfer(TransportMediator.java:56)
>
> at
> org.apache.airavata.mft.agent.MFTAgent.lambda$acceptRequests$6(MFTAgent.java:143)
>
> at java.base/java.util.Optional.ifPresent(Optional.java:176)
>
> at
> org.apache.airavata.mft.agent.MFTAgent.lambda$acceptRequests$7(MFTAgent.java:106)
>
> at 

Re: [airavata-django-portal] Cannot keep the localhost db running for more than half a hour

[DImuthu Upeksha]

Dinuka,

I uploaded new certificates for ide integration module. You have do a full
cleanup of docker containers and restart APIServer to make it
fully functional.

docker-compose down
docker-compose rm
rm -rf database_data
docker-compose up

Here [2] are the commands to generate the self signed certificate in case
if we need it in future.

[2]
https://github.com/apache/airavata/tree/develop/modules/ide-integration#note-optional-creating-certificates-if-expired

Thanks
Dimuthu


On Thu, Apr 9, 2020 at 9:26 PM DImuthu Upeksha 
wrote:

> Dinuka
>
> I saw couple of issues but I believe that below is the closest issue
> from your logs
>
> 2020-04-09 00:00:39,326 [pool-79-thread-1] ERROR
> o.a.a.s.s.KeyCloakSecurityManager Error occurred while checking if user:
> default-admin is authorized for action: /airavata/getGatewayGroups in
> gateway: default []
> javax.net.ssl.SSLHandshakeException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target
> at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:324)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:262)
> at
> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:645)
> at
> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:464)
> at
> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360)
> at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
> at
> java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
> at
> java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
> at
> java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:181)
> at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
> at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1460)
> at
> java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1368)
> at
> java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:437)
> at
> java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
> at
> java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:171)
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)
> at
> java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
> at
> org.apache.airavata.service.security.KeyCloakSecurityManager.getFromUrl(KeyCloakSecurityManager.java:394)
> at
> org.apache.airavata.service.security.KeyCloakSecurityManager.getUserInfo(KeyCloakSecurityManager.java:318)
> at
> org.apache.airavata.service.security.KeyCloakSecurityManager.validateToken(KeyCloakSecurityManager.java:350)
> at
> org.apache.airavata.service.security.KeyCloakSecurityManager.getGatewayGroupMembership(KeyCloakSecurityManager.java:331)
> at
> org.apache.airavata.service.security.KeyCloakSecurityManager.isUserAuthorized(KeyCloakSecurityManager.java:253)
> at
> org.apache.airavata.service.security.interceptor.SecurityInterceptor.authorize(SecurityInterceptor.java:67)
> at
> org.apache.airavata.service.security.interceptor.SecurityInterceptor.invoke(SecurityInterceptor.java:52)
> at
> com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
> at
> com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
> at
> org.apache.airavata.api.server.handler.AiravataServerHandler$$EnhancerByGuice$$11b56453.getGatewayGroups()
> at
> org.apache.airavata.api.Airavata$Processor$getGatewayGroups.getResult(Airavata.java:23303)
> at
> org.apache.airavata.api.Airavata$Processor$getGatewayGroups.getResult(Airavata.java:23287)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.bas

Re: [airavata-django-portal] Cannot keep the localhost db running for more than half a hour

[DImuthu Upeksha]

:6D:5E:7A:95
 SHA256:
F3:A1:53:31:05:1D:F0:E2:2B:55:95:44:3F:6E:AB:AE:75:65:9F:8D:C1:8F:0D:4A:AF:AE:4C:80:BA:45:00:1F
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key

[1]
https://github.com/apache/airavata/tree/master/modules/ide-integration/src/main/resources/keystores

Thanks
Dimuthu



On Thu, Apr 9, 2020 at 8:20 PM Dinuka Desilva 
wrote:

> Hi Dimuthu,
>
> Please find them below.
>
>  settings_local.py
> <https://drive.google.com/file/d/1SAxdZpzptjTAJTkfzMcnq99SCjjQTpWm/view?usp=drive_web>
>
>  Screenshot 2020-04-03 at 4.03.25 PM.png
> <https://drive.google.com/file/d/1AKf_44WB2cFRfPKNqot8dvH6PISg8Itm/view?usp=drive_web>
> Regards,
> Dinuka
>
> On Fri, Apr 10, 2020 at 5:00 AM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com> wrote:
>
>> Hi Dinuka,
>>
>> Can you please resend the screenshot you have sent before? I guess apache
>> mail server drops attachments for some reason. Can you upload it to drive
>> or dropbox and share the link?
>>
>> Thanks
>> Dimuthu
>>
>> On Thu, Apr 9, 2020 at 11:27 AM Dinuka Desilva 
>> wrote:
>>
>>> Hi Dimuthu,
>>>
>>> I'm using the "develop" branch on both repos. (airavata and
>>> airavata-django-portal)
>>>
>>> Regards,
>>> Dinuka
>>>
>>> On Thu, Apr 9, 2020 at 2:15 PM DImuthu Upeksha <
>>> dimuthu.upeks...@gmail.com> wrote:
>>>
>>>> Dinuka,
>>>>
>>>> What is the branch you are using?
>>>>
>>>> Dimuthu
>>>>
>>>> On Wed, Apr 8, 2020 at 2:45 PM Dinuka Desilva <
>>>> l.dinukadesi...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Let me upload the logs of docker-compose, airavata-server and
>>>>> airavata-django-portal below. I've been actually having this issue
>>>>> continuously and not once in a while. Most of the time after every 20-15
>>>>> minutes,I had to recreate everything. Sometimes even the first attempt end
>>>>> up with this issue.
>>>>>
>>>>> Regards,
>>>>> Dinuka
>>>>>
>>>>> On Wed, Apr 8, 2020 at 3:06 AM Christie, Marcus Aaron 
>>>>> wrote:
>>>>>
>>>>>> I've seen this problem too. Are there an errors in the
>>>>>> 'docker-compose up' output?
>>>>>>
>>>>>> On Apr 5, 2020, at 9:55 PM, Isuru Ranawaka 
>>>>>> wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Apr 5, 2020 at 8:59 PM Suresh Marru 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Isuru,
>>>>>>>
>>>>>>> I wonder if it is related to renew tokens? Any insights?
>>>>>>>
>>>>>>
>>>>>> There is a chance. But,  AFAIK  the server caches authorization
>>>>>> decisions for at least one hour through auth cache (by default this is
>>>>>> enabled). So it is unlikely that the server may try to renew tokens 
>>>>>> before
>>>>>> that time. Breaking in 30 minutes interval is wired.  Anyhow, there is a
>>>>>> probability of occurring this if sever unable to access  sharing 
>>>>>> registry.
>>>>>> Hence,   I think better to check whether  DB connections and IAM sever
>>>>>> connections are correctly established.
>>>>>>
>>>>>> Is there any error logs on server-side ?
>>>>>>
>>>>>> thanks
>>>>>> Isuru
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Suresh
>>>>>>>
>>>>>>> On Apr 3, 2020, at 6:48 AM, Dinuka Desilva <
>>>>>>> l.dinukadesi...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I got the airava and airavata-django-portal running on localhost by
>>>>>>> executing following script on 4 terminals one after the other.
>>>>>>>
>>>>>>>1. docker-compose down && rm -rf database_data/ &&
>>>>>>>docker-compose up
>>>>>>>2. mvn exec:java
>>>>>>>
>>>>>>> -Dexec.mainClass="org.apache.airavata.ide.integration.APIServerStarter"
>>>>>>>3. . ./build_js.sh
>>>>>>>4. source ./venv/bin/activate && rm -rf ./tmp/ && rm -rf
>>>>>>>db.sqlite3 && python manage.py migrate && python manage.py
>>>>>>>load_default_gateway && python manage.py runserver
>>>>>>>
>>>>>>>
>>>>>>> With this I could access the localhost server at
>>>>>>> http://localhost:8000. But, only for less than half an hour it
>>>>>>> worked and then it's ended up with the following error. It has the 
>>>>>>> logged
>>>>>>> in user though an error occurred.
>>>>>>>
>>>>>>> 
>>>>>>>
>>>>>>> Neither restarting the servers nor clearing cache didn't solve this
>>>>>>> issue. And I had to execute above four script again which recreates the
>>>>>>> database. For kind of each 20-30 minutes, I had to do this. I'm doubtful
>>>>>>> whether I'm doing anything wrong.
>>>>>>>
>>>>>>> Also, below I have attached the settings_local.py.
>>>>>>>
>>>>>>> Thanka & Regards,
>>>>>>> Dinuka
>>>>>>> 
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Research Software Engineer
>>>>>> Indiana University, IN
>>>>>>
>>>>>>
>>>>>>

Re: [airavata-django-portal] Cannot keep the localhost db running for more than half a hour

[DImuthu Upeksha]

Hi Dinuka,

Can you please resend the screenshot you have sent before? I guess apache
mail server drops attachments for some reason. Can you upload it to drive
or dropbox and share the link?

Thanks
Dimuthu

On Thu, Apr 9, 2020 at 11:27 AM Dinuka Desilva 
wrote:

> Hi Dimuthu,
>
> I'm using the "develop" branch on both repos. (airavata and
> airavata-django-portal)
>
> Regards,
> Dinuka
>
> On Thu, Apr 9, 2020 at 2:15 PM DImuthu Upeksha 
> wrote:
>
>> Dinuka,
>>
>> What is the branch you are using?
>>
>> Dimuthu
>>
>> On Wed, Apr 8, 2020 at 2:45 PM Dinuka Desilva 
>> wrote:
>>
>>> Hi,
>>>
>>> Let me upload the logs of docker-compose, airavata-server and
>>> airavata-django-portal below. I've been actually having this issue
>>> continuously and not once in a while. Most of the time after every 20-15
>>> minutes,I had to recreate everything. Sometimes even the first attempt end
>>> up with this issue.
>>>
>>> Regards,
>>> Dinuka
>>>
>>> On Wed, Apr 8, 2020 at 3:06 AM Christie, Marcus Aaron 
>>> wrote:
>>>
>>>> I've seen this problem too. Are there an errors in the 'docker-compose
>>>> up' output?
>>>>
>>>> On Apr 5, 2020, at 9:55 PM, Isuru Ranawaka  wrote:
>>>>
>>>> Hi all,
>>>>
>>>>
>>>>
>>>> On Sun, Apr 5, 2020 at 8:59 PM Suresh Marru  wrote:
>>>>
>>>>> Hi Isuru,
>>>>>
>>>>> I wonder if it is related to renew tokens? Any insights?
>>>>>
>>>>
>>>> There is a chance. But,  AFAIK  the server caches authorization
>>>> decisions for at least one hour through auth cache (by default this is
>>>> enabled). So it is unlikely that the server may try to renew tokens before
>>>> that time. Breaking in 30 minutes interval is wired.  Anyhow, there is a
>>>> probability of occurring this if sever unable to access  sharing registry.
>>>> Hence,   I think better to check whether  DB connections and IAM sever
>>>> connections are correctly established.
>>>>
>>>> Is there any error logs on server-side ?
>>>>
>>>> thanks
>>>> Isuru
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Suresh
>>>>>
>>>>> On Apr 3, 2020, at 6:48 AM, Dinuka Desilva 
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I got the airava and airavata-django-portal running on localhost by
>>>>> executing following script on 4 terminals one after the other.
>>>>>
>>>>>1. docker-compose down && rm -rf database_data/ && docker-compose
>>>>>up
>>>>>2. mvn exec:java
>>>>>-Dexec.mainClass="org.apache.airavata.ide.integration.APIServerStarter"
>>>>>3. . ./build_js.sh
>>>>>4. source ./venv/bin/activate && rm -rf ./tmp/ && rm -rf
>>>>>db.sqlite3 && python manage.py migrate && python manage.py
>>>>>load_default_gateway && python manage.py runserver
>>>>>
>>>>>
>>>>> With this I could access the localhost server at http://localhost:8000.
>>>>> But, only for less than half an hour it worked and then it's ended up with
>>>>> the following error. It has the logged in user though an error occurred.
>>>>>
>>>>> 
>>>>>
>>>>> Neither restarting the servers nor clearing cache didn't solve this
>>>>> issue. And I had to execute above four script again which recreates the
>>>>> database. For kind of each 20-30 minutes, I had to do this. I'm doubtful
>>>>> whether I'm doing anything wrong.
>>>>>
>>>>> Also, below I have attached the settings_local.py.
>>>>>
>>>>> Thanka & Regards,
>>>>> Dinuka
>>>>> 
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Research Software Engineer
>>>> Indiana University, IN
>>>>
>>>>
>>>>

Re: [airavata-django-portal] Cannot keep the localhost db running for more than half a hour

[DImuthu Upeksha]

Dinuka,

What is the branch you are using?

Dimuthu

On Wed, Apr 8, 2020 at 2:45 PM Dinuka Desilva 
wrote:

> Hi,
>
> Let me upload the logs of docker-compose, airavata-server and
> airavata-django-portal below. I've been actually having this issue
> continuously and not once in a while. Most of the time after every 20-15
> minutes,I had to recreate everything. Sometimes even the first attempt end
> up with this issue.
>
> Regards,
> Dinuka
>
> On Wed, Apr 8, 2020 at 3:06 AM Christie, Marcus Aaron 
> wrote:
>
>> I've seen this problem too. Are there an errors in the 'docker-compose
>> up' output?
>>
>> On Apr 5, 2020, at 9:55 PM, Isuru Ranawaka  wrote:
>>
>> Hi all,
>>
>>
>>
>> On Sun, Apr 5, 2020 at 8:59 PM Suresh Marru  wrote:
>>
>>> Hi Isuru,
>>>
>>> I wonder if it is related to renew tokens? Any insights?
>>>
>>
>> There is a chance. But,  AFAIK  the server caches authorization decisions
>> for at least one hour through auth cache (by default this is enabled). So
>> it is unlikely that the server may try to renew tokens before that time.
>> Breaking in 30 minutes interval is wired.  Anyhow, there is a probability
>> of occurring this if sever unable to access  sharing registry. Hence,   I
>> think better to check whether  DB connections and IAM sever connections are
>> correctly established.
>>
>> Is there any error logs on server-side ?
>>
>> thanks
>> Isuru
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>>
>>> Suresh
>>>
>>> On Apr 3, 2020, at 6:48 AM, Dinuka Desilva 
>>> wrote:
>>>
>>> Hi,
>>>
>>> I got the airava and airavata-django-portal running on localhost by
>>> executing following script on 4 terminals one after the other.
>>>
>>>1. docker-compose down && rm -rf database_data/ && docker-compose up
>>>2. mvn exec:java
>>>-Dexec.mainClass="org.apache.airavata.ide.integration.APIServerStarter"
>>>3. . ./build_js.sh
>>>4. source ./venv/bin/activate && rm -rf ./tmp/ && rm -rf db.sqlite3
>>>&& python manage.py migrate && python manage.py load_default_gateway &&
>>>python manage.py runserver
>>>
>>>
>>> With this I could access the localhost server at http://localhost:8000.
>>> But, only for less than half an hour it worked and then it's ended up with
>>> the following error. It has the logged in user though an error occurred.
>>>
>>> 
>>>
>>> Neither restarting the servers nor clearing cache didn't solve this
>>> issue. And I had to execute above four script again which recreates the
>>> database. For kind of each 20-30 minutes, I had to do this. I'm doubtful
>>> whether I'm doing anything wrong.
>>>
>>> Also, below I have attached the settings_local.py.
>>>
>>> Thanka & Regards,
>>> Dinuka
>>> 
>>>
>>>
>>>
>>
>> --
>> Research Software Engineer
>> Indiana University, IN
>>
>>
>>

Re: Apache Airavata MFT - AWS/GCS support

[DImuthu Upeksha]

Aravind,

Here [2] is the complete commit for S3 transport implementation but don't
get confused by the amount of changes as this includes both transport
implementation and the service backend implementations. If you need to
implement a new transport, you need to implement a Receiver, Sender and a
MetadataCollector like this [3]. Then you need to add that resource support
to Resource service and Secret service [4] [5]. You can similarly do that
for Azure. A sample SCP -> S3 transfer request is like below. Hope that
helps.

String sourceId = "remote-ssh-resource";
String sourceToken = "local-ssh-cred";
String sourceType = "SCP";
String destId = "s3-file";
String destToken = "s3-cred";
String destType = "S3";

TransferApiRequest request = TransferApiRequest.newBuilder()
.setSourceId(sourceId)
.setSourceToken(sourceToken)
.setSourceType(sourceType)
.setDestinationId(destId)
.setDestinationToken(destToken)
.setDestinationType(destType)
.setAffinityTransfer(false).build();


[2]
https://github.com/apache/airavata-mft/commit/62fae3d0ab2921fa8bf0bea7970e233f842e6948
[3]
https://github.com/apache/airavata-mft/tree/master/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3
[4]
https://github.com/apache/airavata-mft/blob/master/services/resource-service/stub/src/main/proto/ResourceService.proto#L90
[5]
https://github.com/apache/airavata-mft/blob/master/services/secret-service/stub/src/main/proto/SecretService.proto#L45

Thanks
Dimuthu


On Sun, Apr 5, 2020 at 12:10 AM DImuthu Upeksha 
wrote:

> There is a working on S3 transport in my local copy. Will commit it once I
> test it out properly. You can follow the same pattern for any cloud
> provider which has clients with streaming IO. Streaming among different
> transfer protocols inside an Agent has been discussed in the last part of
> this [1] document. Try to get the conceptual idea from that and reverse
> engineer SCP transport.
>
> [1]
> https://docs.google.com/document/d/1zrO4Z1dn7ENhm1RBdVCw-dDpWiebaZEWy66ceTWoOlo
>
> Dimuthu
>
> On Sat, Apr 4, 2020 at 9:22 PM Aravind Ramalingam 
> wrote:
>
>> Hello,
>>
>> We were looking at the existing code in the project. We could find
>> implementations only for local copy and SCP.
>> We were confused on how to go about with an external provider like S3 or
>> Azure? Since it would require integrating with their respective clients.
>>
>> Thank you
>> Aravind Ramalingam
>>
>> > On Apr 4, 2020, at 21:15, Suresh Marru  wrote:
>> >
>> > Hi Aravind,
>> >
>> > I have to catch up with the code, but you may want to look at the S3
>> implementation and extend it to Azure, GCP or other cloud services like
>> Box, Dropbox and so on.
>> >
>> > There could be many use cases, here is an idea:
>> >
>> > * Compute a job on a supercomputer with SCP access and push the outputs
>> to a Cloud storage.
>> >
>> > Suresh
>> >
>> >> On Apr 4, 2020, at 8:09 PM, Aravind Ramalingam 
>> wrote:
>> >>
>> >> Hello,
>> >>
>> >> We set up the MFT project on local system and tested out SCP transfer
>> between JetStream VMs, we were wondering how the support can be extended
>> for AWS/GCS.
>> >>
>> >> As per our understanding, the current implementation has support for
>> two protocols i.e. local-transport and scp-transport. Would we have to
>> modify/add to the code base to extend support for AWS/GCS clients?
>> >>
>> >> Could you please provide suggestions for this use case.
>> >>
>> >> Thank you
>> >> Aravind Ramalingam
>> >
>>
>

Re: Question regarding Resource and Secret service Backend in Production

[DImuthu Upeksha]

Adding Isuru explicitly to the discussion

Good question. This is something that we need to properly evaluate the
margin of integration between Custos and general Airavata data models. I
agree that resources are not directly integrated with Custos but at some
point, Resource Service has to talk to Custos to check for the permissions
for accessing a particular resource. Based on current standing, I would say
resource backend will still talk to Airavata backend class and  within
Airavata backend, it will talk to Custos only to check for authorization.

Isuru has been doing a lot of exploration in this area and he might have a
better explanation than mine.

Thanks

On Sat, Apr 4, 2020 at 9:16 PM Aravind Ramalingam  wrote:

> Hello,
>
> As per my current understanding wouldn't the Custos project only act as a
> production set up for secret service? To store our API keys or SSH keys.
>
> Wouldn't we require something additional for the resource service where we
> would store our resource information?
>
> Thank you
> Aravind Ramalingam
>
> On Apr 4, 2020, at 21:07, Suresh Marru  wrote:
>
> Hi Akshay,
>
> For production deployments I suggest to integrate with Airavata Custos
> services - https://github.com/apache/airavata-custos/tree/develop
>
> Please ask questions as you explore these.
>
> Suresh
>
> On Apr 4, 2020, at 7:29 PM, Rajvanshi, Akshay  wrote:
>
> Hello,
>
> Currently we did the local setup for the airavata-mft and copied a file
> within our VM, but currently we have an implementation for resource and
> service backend with file based backend. How do we go about implementing at
> the production level ? Are we expected to add the support for the service
> we plan to integrate with separately?
>
> Thanks
>
>
>

Re: Apache Airavata MFT - AWS/GCS support

[DImuthu Upeksha]

There is a working on S3 transport in my local copy. Will commit it once I
test it out properly. You can follow the same pattern for any cloud
provider which has clients with streaming IO. Streaming among different
transfer protocols inside an Agent has been discussed in the last part of
this [1] document. Try to get the conceptual idea from that and reverse
engineer SCP transport.

[1]
https://docs.google.com/document/d/1zrO4Z1dn7ENhm1RBdVCw-dDpWiebaZEWy66ceTWoOlo

Dimuthu

On Sat, Apr 4, 2020 at 9:22 PM Aravind Ramalingam  wrote:

> Hello,
>
> We were looking at the existing code in the project. We could find
> implementations only for local copy and SCP.
> We were confused on how to go about with an external provider like S3 or
> Azure? Since it would require integrating with their respective clients.
>
> Thank you
> Aravind Ramalingam
>
> > On Apr 4, 2020, at 21:15, Suresh Marru  wrote:
> >
> > Hi Aravind,
> >
> > I have to catch up with the code, but you may want to look at the S3
> implementation and extend it to Azure, GCP or other cloud services like
> Box, Dropbox and so on.
> >
> > There could be many use cases, here is an idea:
> >
> > * Compute a job on a supercomputer with SCP access and push the outputs
> to a Cloud storage.
> >
> > Suresh
> >
> >> On Apr 4, 2020, at 8:09 PM, Aravind Ramalingam 
> wrote:
> >>
> >> Hello,
> >>
> >> We set up the MFT project on local system and tested out SCP transfer
> between JetStream VMs, we were wondering how the support can be extended
> for AWS/GCS.
> >>
> >> As per our understanding, the current implementation has support for
> two protocols i.e. local-transport and scp-transport. Would we have to
> modify/add to the code base to extend support for AWS/GCS clients?
> >>
> >> Could you please provide suggestions for this use case.
> >>
> >> Thank you
> >> Aravind Ramalingam
> >
>

Setting up MFT in local envornments

[DImuthu Upeksha]

Hi folks

This is specially for GSoC applicants who started working in Airavata
Managed File Transfer project. First of all I should apologize for the
confusion happened in setting up MFT in your local environments. This a
very new project and we are working on specific goals internally so some
parts of developer adaptation are yet to be done but we are doing our best
to reduce the confusion.

Now you have a much better detailed installation guideline in the readme
[1] with a sample and  file based backends for both Resource and Secret
services so that you can easily configure resources and credentials for
development purposes. You can switch backends by changing the
applicationContext.xml [2] [3] at anytime.

[1] https://github.com/apache/airavata-mft#building-from-the-script
[2]
https://github.com/apache/airavata-mft/blob/master/services/resource-service/server/src/main/resources/applicationContext.xml
[3]
https://github.com/apache/airavata-mft/blob/master/services/secret-service/server/src/main/resources/applicationContext.xml

Thanks
Dimuthu

Re: [External] Airavata MFT project errors

[DImuthu Upeksha]

Hi Aravind,

Yes this is confusing. These configurations are for Airavata backend (in
future for custos backend). I prefixed these configurations by respective
backend name [1]. I will shortly send a detailed note to dev list on how to
setup MFT properly in local environments. Let's move the discussion to
there as it would be helpful for general public.

[1]
https://github.com/apache/airavata-mft/blob/master/services/resource-service/server/src/main/resources/application.properties#L23

Thanks
Dimuthu


On Thu, Apr 2, 2020 at 5:28 PM Aravind Ramalingam  wrote:

> Hello,
>
> Thank you, that makes sense!
> But I was actually wondering what the current code base is doing? Since in
> the application.properties file for resource service, there is a specific
> address for a registry server.
> [image: image.png]
> Are we connecting to this for getting resource information?
>
> Also, in Secret service the application.properties file has localhost for
> credentials server.
> [image: image.png]
> Are we supposed to run any other project to support the service?
>
> Thank you
> Aravind Ramalingam
>
> On Thu, Apr 2, 2020 at 4:04 PM DImuthu Upeksha 
> wrote:
>
>> Aravind,
>>
>> Resource and Secret services are only proxy services. It can bind to any
>> backend type. Custos is one backend. In production use, it should bind to
>> custos. But in local development, it's inconvenient to work with multiple
>> moving parts so in that case it can bind to a local backend which receives
>> data through couple of files where you can change content easily.
>>
>> Dimuthu
>>
>> On Thu, Apr 2, 2020 at 3:56 PM Aravind Ramalingam 
>> wrote:
>>
>>> Hi Dimuthu,
>>>
>>> Thank you for that.
>>> So you mean to say that the resource service and secret service
>>> currently have the data predefined and that is why the proto message has
>>> default numbers as it's values?
>>> A small question on that, are the data stored internally or do the
>>> resource service and secret service connect to an external server?
>>> Cause I initially thought the secret service will connect to the Custos
>>> project for it's data.
>>>
>>> Thank you
>>> Aravind Ramalingam
>>>
>>> On Apr 2, 2020, at 15:24, DImuthu Upeksha 
>>> wrote:
>>>
>>> 
>>> Hi Aravind,
>>>
>>> Yes your understanding is correct. We assume that Resource service and
>>> Secret Service has all the data you need to initiate the transfer. I'm
>>> working on a standalone version of MFT where you can configure credentials
>>> and resources as a configuration file for development purposes and
>>> hopefully I will be able to push it today. Once it's done, you can use them
>>> to test the framework. With that, I will submit some samples as well.
>>>
>>> Thanks
>>> Dimuthu
>>>
>>> On Thu, Apr 2, 2020 at 1:01 PM Aravind Ramalingam 
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> Thank you for clarifying.
>>>> If I've understood the architecture correctly, even if we are using
>>>> another language we would require to build the proto files only for API? As
>>>> the other services are internal to the system.
>>>>
>>>> Could you please provide a sample message that can be passed to the
>>>> service? It would really help us get started.
>>>>
>>>> Thank you
>>>> Aravind Ramalingam
>>>>
>>>> On Apr 2, 2020, at 12:26, DImuthu Upeksha 
>>>> wrote:
>>>>
>>>> 
>>>> Hi Aravind,
>>>>
>>>> Currently all the calls to the API are performed through gRPC. For
>>>> Java, clients are already generated from the maven build, but if you are
>>>> using any other language, you have to build it using proto files in the
>>>> stub module of each service. Usability is a place where we need to improve
>>>> in MFT as we are currently supporting it only to bind to an existing
>>>> service that require MFT rather than a user directly talking to it. But it
>>>> is also a valid use case and we need to improve it.
>>>>
>>>> Thanks
>>>> Dimuthu
>>>>
>>>> On Thu, Apr 2, 2020 at 4:29 AM Aravind Ramalingam 
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Thank you for the solution. That resolved the dependencies issue.
>>>>> After that i started the services in the order mentioned

Re: [External] Airavata MFT project errors

[DImuthu Upeksha]

Aravind,

Resource and Secret services are only proxy services. It can bind to any
backend type. Custos is one backend. In production use, it should bind to
custos. But in local development, it's inconvenient to work with multiple
moving parts so in that case it can bind to a local backend which receives
data through couple of files where you can change content easily.

Dimuthu

On Thu, Apr 2, 2020 at 3:56 PM Aravind Ramalingam  wrote:

> Hi Dimuthu,
>
> Thank you for that.
> So you mean to say that the resource service and secret service currently
> have the data predefined and that is why the proto message has default
> numbers as it's values?
> A small question on that, are the data stored internally or do the
> resource service and secret service connect to an external server?
> Cause I initially thought the secret service will connect to the Custos
> project for it's data.
>
> Thank you
> Aravind Ramalingam
>
> On Apr 2, 2020, at 15:24, DImuthu Upeksha 
> wrote:
>
> 
> Hi Aravind,
>
> Yes your understanding is correct. We assume that Resource service and
> Secret Service has all the data you need to initiate the transfer. I'm
> working on a standalone version of MFT where you can configure credentials
> and resources as a configuration file for development purposes and
> hopefully I will be able to push it today. Once it's done, you can use them
> to test the framework. With that, I will submit some samples as well.
>
> Thanks
> Dimuthu
>
> On Thu, Apr 2, 2020 at 1:01 PM Aravind Ramalingam 
> wrote:
>
>> Hello,
>>
>> Thank you for clarifying.
>> If I've understood the architecture correctly, even if we are using
>> another language we would require to build the proto files only for API? As
>> the other services are internal to the system.
>>
>> Could you please provide a sample message that can be passed to the
>> service? It would really help us get started.
>>
>> Thank you
>> Aravind Ramalingam
>>
>> On Apr 2, 2020, at 12:26, DImuthu Upeksha 
>> wrote:
>>
>> 
>> Hi Aravind,
>>
>> Currently all the calls to the API are performed through gRPC. For Java,
>> clients are already generated from the maven build, but if you are using
>> any other language, you have to build it using proto files in the stub
>> module of each service. Usability is a place where we need to improve in
>> MFT as we are currently supporting it only to bind to an existing service
>> that require MFT rather than a user directly talking to it. But it is also
>> a valid use case and we need to improve it.
>>
>> Thanks
>> Dimuthu
>>
>> On Thu, Apr 2, 2020 at 4:29 AM Aravind Ramalingam 
>> wrote:
>>
>>> Hi,
>>>
>>> Thank you for the solution. That resolved the dependencies issue.
>>> After that i started the services in the order mentioned in the README.
>>>
>>> 
>>>
>>>
>>> The services started successfully and connected the external Consul
>>> host.
>>> But I am confused as to how a user would make a call to the
>>> ApiServiceApplication? Are there any endpoints exposed or is it only
>>> through gRPC calls?
>>> A little more insight would be helpful.
>>>
>>> Thank you,
>>> Aravind Ramalingam
>>>
>>> On Thu, Apr 2, 2020 at 12:44 AM Bandaru, Vivek Shresta 
>>> wrote:
>>>
>>>> Hi Aravind,
>>>>
>>>>
>>>>
>>>> This seems to be an Intellij issue. I resolved the dependency issues on
>>>> my Intellij by enabling ‘Use plugin registry’ in maven preferences and
>>>> rebuilding my project.
>>>>
>>>>
>>>>
>>>> Seems like there are 5 SpringBoot microservices in MFT.
>>>>
>>>>
>>>>
>>>> 
>>>>
>>>>
>>>>
>>>> As far as my understanding goes, MFT alone doesn’t do anything, it’s a
>>>> helper service. It works along with ‘Airavata’ server.
>>>>
>>>>
>>>>
>>>> If someone can point us towards some MFT Tutorials and documentation on
>>>> how to setup Airavata along with MFT, that would be really helpful.
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Vivek.
>>>>
>>>>
>>>>
>>>> *From: *Aravind Ramalingam 
>>>> *Reply-To: *"dev@airavata.apache.org" 
>>>> *Date: *Wednesday, April 1, 2020 at 9:41 PM
>>>> *To: *"dev@airavata.apache.org" 
>>>> *Subject: *[External] Airavata MFT project errors
>>>>
>>>>
>>>>
>>>> This message was sent from a non-IU address. Please exercise caution
>>>> when clicking links or opening attachments from external sources.
>>>>
>>>>
>>>>
>>>> Hello,
>>>>
>>>>
>>>>
>>>> I am trying to compile and run the Airvata MFT project.
>>>>
>>>> After completing mvn clean install successfully, I found that the
>>>> services are not running due to errors in certain files.
>>>>
>>>>
>>>>
>>>> 
>>>>
>>>>
>>>>
>>>> I have attached the screenshot for your reference. I suspect its some
>>>> missing dependencies. Can you please help me resolve this?
>>>>
>>>> Also can you guide me on the sequence of how to start the applications,
>>>> i am confused how to run the project.
>>>>
>>>>
>>>>
>>>> Thank you
>>>>
>>>> Aravind Ramalingam
>>>>
>>>

Re: [External] Airavata MFT project errors

[DImuthu Upeksha]

Hi Aravind,

Yes your understanding is correct. We assume that Resource service and
Secret Service has all the data you need to initiate the transfer. I'm
working on a standalone version of MFT where you can configure credentials
and resources as a configuration file for development purposes and
hopefully I will be able to push it today. Once it's done, you can use them
to test the framework. With that, I will submit some samples as well.

Thanks
Dimuthu

On Thu, Apr 2, 2020 at 1:01 PM Aravind Ramalingam  wrote:

> Hello,
>
> Thank you for clarifying.
> If I've understood the architecture correctly, even if we are using
> another language we would require to build the proto files only for API? As
> the other services are internal to the system.
>
> Could you please provide a sample message that can be passed to the
> service? It would really help us get started.
>
> Thank you
> Aravind Ramalingam
>
> On Apr 2, 2020, at 12:26, DImuthu Upeksha 
> wrote:
>
> 
> Hi Aravind,
>
> Currently all the calls to the API are performed through gRPC. For Java,
> clients are already generated from the maven build, but if you are using
> any other language, you have to build it using proto files in the stub
> module of each service. Usability is a place where we need to improve in
> MFT as we are currently supporting it only to bind to an existing service
> that require MFT rather than a user directly talking to it. But it is also
> a valid use case and we need to improve it.
>
> Thanks
> Dimuthu
>
> On Thu, Apr 2, 2020 at 4:29 AM Aravind Ramalingam 
> wrote:
>
>> Hi,
>>
>> Thank you for the solution. That resolved the dependencies issue.
>> After that i started the services in the order mentioned in the README.
>>
>> 
>>
>>
>> The services started successfully and connected the external Consul host.
>> But I am confused as to how a user would make a call to the
>> ApiServiceApplication? Are there any endpoints exposed or is it only
>> through gRPC calls?
>> A little more insight would be helpful.
>>
>> Thank you,
>> Aravind Ramalingam
>>
>> On Thu, Apr 2, 2020 at 12:44 AM Bandaru, Vivek Shresta 
>> wrote:
>>
>>> Hi Aravind,
>>>
>>>
>>>
>>> This seems to be an Intellij issue. I resolved the dependency issues on
>>> my Intellij by enabling ‘Use plugin registry’ in maven preferences and
>>> rebuilding my project.
>>>
>>>
>>>
>>> Seems like there are 5 SpringBoot microservices in MFT.
>>>
>>>
>>>
>>> 
>>>
>>>
>>>
>>> As far as my understanding goes, MFT alone doesn’t do anything, it’s a
>>> helper service. It works along with ‘Airavata’ server.
>>>
>>>
>>>
>>> If someone can point us towards some MFT Tutorials and documentation on
>>> how to setup Airavata along with MFT, that would be really helpful.
>>>
>>>
>>>
>>> Regards,
>>>
>>> Vivek.
>>>
>>>
>>>
>>> *From: *Aravind Ramalingam 
>>> *Reply-To: *"dev@airavata.apache.org" 
>>> *Date: *Wednesday, April 1, 2020 at 9:41 PM
>>> *To: *"dev@airavata.apache.org" 
>>> *Subject: *[External] Airavata MFT project errors
>>>
>>>
>>>
>>> This message was sent from a non-IU address. Please exercise caution
>>> when clicking links or opening attachments from external sources.
>>>
>>>
>>>
>>> Hello,
>>>
>>>
>>>
>>> I am trying to compile and run the Airvata MFT project.
>>>
>>> After completing mvn clean install successfully, I found that the
>>> services are not running due to errors in certain files.
>>>
>>>
>>>
>>> 
>>>
>>>
>>>
>>> I have attached the screenshot for your reference. I suspect its some
>>> missing dependencies. Can you please help me resolve this?
>>>
>>> Also can you guide me on the sequence of how to start the applications,
>>> i am confused how to run the project.
>>>
>>>
>>>
>>> Thank you
>>>
>>> Aravind Ramalingam
>>>
>>

Re: MFT and data access for running jobs

[DImuthu Upeksha]

Copying data periodically to gateway storage/ user's desktop can be done
from MFT if we have following

1. If it's copying to gateway storage, gateway storage side should have a
MFT Agent installed
2. If it's for user's desktop, user should have MFT Agent installed and
provide write access to a particular directory.
3. However in both cases, we need to have another service instructing to
MFT when, what, where to copy at each iteration.
4. In addition to that, we need some changes to Airavata API models to
store and configure metadata for periodic synchronizations.

I believe this is a good GSoC project if someone is willing to take on and
I would like to act as a mentor.

Dimuthu

On Thu, Mar 26, 2020 at 12:47 PM Pamidighantam, Sudhakar 
wrote:

> I am not suggesting we mount any disk but potentially transfer the data in
> that remote HPC disk to storage in the gateway and provide access from
> gateway storage or directly to users desktop periodically/or on prompt
>  during the run.
>
>
>
> Thanks,
>
> Sudhakar.
>
>
>
> *From: *DImuthu Upeksha 
> *Reply-To: *"dev@airavata.apache.org" 
> *Date: *Thursday, March 26, 2020 at 12:42 PM
> *To: *Airavata Dev 
> *Subject: *Re: MFT and data access for running jobs
>
>
>
> Sudhakar,
>
>
>
> What you are asking is not a direct MFT use case. It's more like a NFS
> mount of a remote file system to a local file system. MFT is mainly
> focussing on handling the data transfer path not synching data between two
> endpoints at realtime.
>
>
>
> Thanks
>
> Dimuthu
>
>
>
> On Thu, Mar 26, 2020 at 12:29 PM Pamidighantam, Sudhakar 
> wrote:
>
> Dimuthu:
>
>
>
> Yes, the working directory on remote HPC cluster.
>
>
>
> The workflow may look like this..
>
>
>
> The user launches a job..
>
> The remote working directory, dynamically defined by Airavata during the
> launch of the experiment is registered as a remote disk accessible
>
> The contents are made available readonly for  users to read/download
>
> Remove this as accessible when the experiment ends
>
> Continue with the rest of the Helix tasks
>
> …
>
>
>
>
>
> Thanks,
>
> Sudhakar.
>
>
>
> *From: *DImuthu Upeksha 
> *Reply-To: *"dev@airavata.apache.org" 
> *Date: *Thursday, March 26, 2020 at 12:23 PM
> *To: *Airavata Dev 
> *Subject: *Re: MFT and data access for running jobs
>
>
>
> Sudhakar,
>
> I’m not sure whether I grabbed your point about this remote working
> directory correctly. Are you taking about the working directory of the
> cluster? Can you please explain the workflow with more details?
>
> Thanks
> Dimuthu
>
>
>
> On Thu, Mar 26, 2020 at 10:21 AM Pamidighantam, Sudhakar 
> wrote:
>
> Dimuthu:
>
>
>
> When the MFT becomes available would there be a way to define the remote
> working directory as a device to provide access to the data there.
>
> You know this has been a long standing need for particularly long running
> jobs.
>
>
>
> Thanks,
>
> Sudhakar.
>
>
>
>

Re: MFT and data access for running jobs

[DImuthu Upeksha]

Sudhakar,

What you are asking is not a direct MFT use case. It's more like a NFS
mount of a remote file system to a local file system. MFT is mainly
focussing on handling the data transfer path not synching data between two
endpoints at realtime.

Thanks
Dimuthu

On Thu, Mar 26, 2020 at 12:29 PM Pamidighantam, Sudhakar 
wrote:

> Dimuthu:
>
>
>
> Yes, the working directory on remote HPC cluster.
>
>
>
> The workflow may look like this..
>
>
>
> The user launches a job..
>
> The remote working directory, dynamically defined by Airavata during the
> launch of the experiment is registered as a remote disk accessible
>
> The contents are made available readonly for  users to read/download
>
> Remove this as accessible when the experiment ends
>
> Continue with the rest of the Helix tasks
>
> …
>
>
>
>
>
> Thanks,
>
> Sudhakar.
>
>
>
> *From: *DImuthu Upeksha 
> *Reply-To: *"dev@airavata.apache.org" 
> *Date: *Thursday, March 26, 2020 at 12:23 PM
> *To: *Airavata Dev 
> *Subject: *Re: MFT and data access for running jobs
>
>
>
> Sudhakar,
>
> I’m not sure whether I grabbed your point about this remote working
> directory correctly. Are you taking about the working directory of the
> cluster? Can you please explain the workflow with more details?
>
> Thanks
> Dimuthu
>
>
>
> On Thu, Mar 26, 2020 at 10:21 AM Pamidighantam, Sudhakar 
> wrote:
>
> Dimuthu:
>
>
>
> When the MFT becomes available would there be a way to define the remote
> working directory as a device to provide access to the data there.
>
> You know this has been a long standing need for particularly long running
> jobs.
>
>
>
> Thanks,
>
> Sudhakar.
>
>
>
>

Re: MFT and data access for running jobs

[DImuthu Upeksha]

Sudhakar,

I’m not sure whether I grabbed your point about this remote working
directory correctly. Are you taking about the working directory of the
cluster? Can you please explain the workflow with more details?

Thanks
Dimuthu

On Thu, Mar 26, 2020 at 10:21 AM Pamidighantam, Sudhakar 
wrote:

> Dimuthu:
>
>
>
> When the MFT becomes available would there be a way to define the remote
> working directory as a device to provide access to the data there.
>
> You know this has been a long standing need for particularly long running
> jobs.
>
>
>
> Thanks,
>
> Sudhakar.
>
>
>

Re: Upgrade Java Version

[DImuthu Upeksha]

Hi Folks,

We have upgraded java version to 11 with OpenJDK in development branch and
successfully deployed it in development environment. Following changes were
performed in the process

1. Upgrades of compiler plugin (3.5.1 -> 3.8.0) , surefire plugin (2.18.1
-> 3.0.0-M1), open jpa (2.4.3 -> 3.1.0). See PR 236 [3]
2. Adding javax dependencies explicitly as they were discontinued in
default java 11 releases. See PR 234 [4]
3. Updating JDK to OpenJDK 11 [5]
4. Disabling Dozer mapper setter optimization [6]. This is a temporary fix
and waiting for dozer team [7] to properly fix this.
5. Fixing remote debug [8]
6. Minor test failure fixes [9]
7. Upgrading Confluent/ Kafka version to (4.1.0 -> 5.3.0) [10]

Travis build and develop environment are now stable :)

Kudos to Isuru, Marcus and Suresh for pulling this off.

[3] https://github.com/apache/airavata/pull/236/files
[4] https://github.com/apache/airavata/pull/234/files
[5]
https://github.com/apache/airavata/commit/fd56623e303e579d7974b721ecd7f931498ad502
[6]
https://github.com/DImuthuUpe/dozer/commit/4d33729e866982cb7cee61065f0719c9c1212334
[7]
https://groups.google.com/forum/?fromgroups=#!topic/dozer-mapper/c1OPee1DwTw
[8]
https://github.com/apache/airavata/commit/5fb5629dfaaf503e1875737c58f703e5cdfad255
[9]
https://github.com/apache/airavata/commit/7b1a2b3525f93a7057325b9df1e7251282e5294f
[10]
https://github.com/apache/airavata/commit/0859211fd8bfb8e2168e95d711fd27d83133c939

Thanks
Dimuthu

On Tue, Oct 22, 2019 at 1:49 PM Suresh Marru  wrote:

> Hi All,
>
> I posted a question for Apache Legal -
> https://issues.apache.org/jira/browse/LEGAL-484
>
> + 1 for moving to Java 11 and get the Travis build working again. We
> probably should try it against both OpenJDK and Oracle JDK latest 11
> releases.
>
> Suresh
>
> On Oct 21, 2019, at 4:34 PM, Christie, Marcus Aaron 
> wrote:
>
> Might be good to ask Apache legal, but it's not like we're distributing
> Oracle JDK, so not sure what kind of guidance they could give.
>
> In the meantime, it would be great to get the Travis build working again.
>
> On Oct 21, 2019, at 4:07 PM, DImuthu Upeksha 
> wrote:
>
> Hi Folks,
>
> What should we do about this? Should we contact Apache legal support to
> get things clarified about the licensing? We might need to stick to a
> decision before next release.
>
> Thanks
> Dimuthu
>
>
>
>

Re: Task is running periodically

[DImuthu Upeksha]

Hi Lee,

I use default configurations for rebalancing. However is this patch
available on 0.9.1?

Thanks
Dimuthu

On Mon, Nov 25, 2019 at 2:19 PM Hunter Lee  wrote:

> Hi Dimuthu,
>
> Are you using periodicalRebalance? That might trigger a rebalance
> periodically even if there's no activity in the cluster.
>
> As for the ConcurrentModification exception, I believe there was a patch
> for that going from 0.8.2 to the latest on the upstream.
>
> Hunter
>
> On Mon, Nov 25, 2019 at 10:13 AM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com> wrote:
>
>> Hi Folks,
>>
>> We have noticed a task in Helix cluster was running periodically even
>> though it was completed at each run. When we look at the logs of the
>> controller, I can see some Concurrent Modification exceptions [1]. However
>> this is a very rare occurrence. We have been using current Helix version
>> for few months on our production deployments but this is the first time we
>> have seen this behavior.
>>
>> Do you have any insight into this?
>>
>> We are using helix 0.8.2
>>
>> [1] https://gist.github.com/DImuthuUpe/33db7bbe5d53fcc38dfc66eb0d45df55
>>
>> Thanks
>> Dimuthu
>>
>

Task is running periodically

[DImuthu Upeksha]

Hi Folks,

We have noticed a task in Helix cluster was running periodically even
though it was completed at each run. When we look at the logs of the
controller, I can see some Concurrent Modification exceptions [1]. However
this is a very rare occurrence. We have been using current Helix version
for few months on our production deployments but this is the first time we
have seen this behavior.

Do you have any insight into this?

We are using helix 0.8.2

[1] https://gist.github.com/DImuthuUpe/33db7bbe5d53fcc38dfc66eb0d45df55

Thanks
Dimuthu

Re: Upgrade Java Version

[DImuthu Upeksha]

Hi Folks,

What should we do about this? Should we contact Apache legal support to get
things clarified about the licensing? We might need to stick to a decision
before next release.

Thanks
Dimuthu

Re: 0.18 Release Planning

[DImuthu Upeksha]

Hi Suresh,

I don't have any objection to create the 0.18 branch now. As the production
deployments are stable, I don't hope that it will require any major updates
in near future. This will even give us enough buffer to stabilize master
after merging the staging.

Thanks
Dimuthu

On Mon, Oct 21, 2019 at 3:34 PM Suresh Marru  wrote:

> Hi All,
>
> Can we create a 0.18 branch from master and start release process? I can
> work on this release in 10 days, but unless there are objections, we can
> immediately create this 0.18 branch and after that merge staging branch
> into master.
>
> Suresh

Re: Upgrade Java Version

[DImuthu Upeksha]

Will this [2] be an issue for future releases? Seems like Oracle JDK/ JRE
11 no longer comes free for production deployments. I'm not sure how it
affects for open source projects

[2]
https://www.oracle.com/technetwork/java/javase/downloads/jdk11-downloads-5066655.html

Thanks
Dimuthu

On Fri, Sep 13, 2019 at 11:58 AM Christie, Marcus Aaron 
wrote:

> +1 for Java 11
>
> On Sep 12, 2019, at 10:28 AM, Suresh Marru  wrote:
>
> + 1 for Java 11.
>
> Any objections to move forward to make Java 11 minimum requirement?
>
> Suresh
>
> On Sep 10, 2019, at 6:06 PM, DImuthu Upeksha 
> wrote:
>
> Hi Suresh,
>
> I would vote for Java 11 as it's relatively mature and a LTS version [1]
>
> [1] https://www.oracle.com/technetwork/java/java-se-support-roadmap.html
>
> Thanks
> Dimuthu
>
> On Thu, Sep 5, 2019 at 8:34 AM Suresh Marru  wrote:
>
>> Hi All,
>>
>> I noticed a travis build fail which complains minimum have required of 9
>> to 12 - “Expected feature release number in range of 9 to 14, but got: 8”.
>>
>> Any thoughts on which version of JDK we should upgrade Airavata to before
>> next release?
>>
>> Cheers,
>> Suresh
>
>
>
>

Re: Upgrade Java Version

[DImuthu Upeksha]

Hi Suresh,

I would vote for Java 11 as it's relatively mature and a LTS version [1]

[1] https://www.oracle.com/technetwork/java/java-se-support-roadmap.html

Thanks
Dimuthu

On Thu, Sep 5, 2019 at 8:34 AM Suresh Marru  wrote:

> Hi All,
>
> I noticed a travis build fail which complains minimum have required of 9
> to 12 - “Expected feature release number in range of 9 to 14, but got: 8”.
>
> Any thoughts on which version of JDK we should upgrade Airavata to before
> next release?
>
> Cheers,
> Suresh

Re: Class not found error occurred when executing "registerSample.sh" in airavata-server-0.17/samples

[DImuthu Upeksha]

Hi Isuru,

Are you following any tutorial to get started with Airavata? Can you please
share more details. Some of them are outdated and we are working on making
them better. If you want to get started with Airavata, I would recommend
you to go with IDE Integration [1] module where you can setup all
components in the IDE. If you want to have an idea about concepts of
Airavata, you can refer to this [2] screencast.

[1] https://github.com/apache/airavata/tree/develop/modules/ide-integration
[2[ https://youtu.be/RJaxqvGPMgU?t=807

Thanks
Dimuthu

On Tue, Jun 18, 2019 at 8:37 AM Isuru Ranawaka  wrote:

> Hi all,
>
> The script "registerSample.sh" requires
> "org.apache.airavata.client.samples.RegisterSampleData" class. But it is
> not  packged in the "airavata-client-samples-0.17.jar" under the libs
> folder.
>
> Thanks
> Isuru
>

Re: Clarification on task level timeouts

[DImuthu Upeksha]

Hi Less,

Thanks a lot for the clarification. This helped a lot. Is there a way where
I can set the Builder.setTimeoutPerTask(infinity)? Will setting -1 work
here?

Thanks
Dimuthu

On Wed, Jun 5, 2019 at 12:22 PM Hunter Lee  wrote:

> Hi Dimuthu -
>
> Task Framework uses the abstraction of workflows, jobs, and tasks. With
> that said, suppose you have a job with 10 tasks. You could set a timeout
> for that job containing 10 tasks, or set a timeout for its individual
> tasks.
>
> For example, if you use Builder.setTimeout(2 hours), then the job itself
> will time out after 2 hours if it hasn't completed within that timeframe.
> Note that all ongoing tasks belonging to that job will be timed
> out/cancelled as well.
>
> If you use Builder.setTimeoutPerTask(30 min), then when an each task
> belonging to that job is actually executed, it will have a timeout of 30
> minutes, and if it hasn't completed after 30 minutes, only that task will
> be timed out, not its parent job.
>
> Hope that helps,
> Hunter
>
> On Wed, Jun 5, 2019 at 6:44 AM DImuthu Upeksha 
> wrote:
>
>> Hi Folks,
>>
>> I have seen some tasks are getting cancelled automatically when they are
>> running more than 1 hour. I feel like the controller is doing that based
>> on
>> some sort of a timeout. Can we remove that timeout? I saw 2 api methods
>> that are related to this problem but I don't have a clear Idea on what
>> each
>> one is doing. Can you please help me to clarify these details.
>>
>> JobConfig.Builder.setTimeout (This one has infinite default value)
>>
>> JobConfig.Builder.setTimeoutPerTask (This one has the default value of 1
>> hour)
>>
>> Thanks
>>
>> Dimuthu
>>
>

Clarification on task level timeouts

[DImuthu Upeksha]

Hi Folks,

I have seen some tasks are getting cancelled automatically when they are
running more than 1 hour. I feel like the controller is doing that based on
some sort of a timeout. Can we remove that timeout? I saw 2 api methods
that are related to this problem but I don't have a clear Idea on what each
one is doing. Can you please help me to clarify these details.

JobConfig.Builder.setTimeout (This one has infinite default value)

JobConfig.Builder.setTimeoutPerTask (This one has the default value of 1 hour)

Thanks

Dimuthu

Re: Zookeeper connection errors in Helix Controller

[DImuthu Upeksha]

Now I'm seeing following error in controller log. Restarting the controller
fixed the issue. We are time to time seeing this in controller with zk
connection issues. Is this also something to do with zk client version?

2019-05-31 13:21:46,669 [Thread-0-SendThread(localhost:2181)] WARN
 o.apache.zookeeper.ClientCnxn  - Session 0x16b0ebbee1d000e for server
localhost/127.0.0.1:2181, unexpected error, closing socket connection and
attempting reconnect
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:102)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)

Thanks
Dimuthu

On Fri, May 31, 2019 at 1:14 PM DImuthu Upeksha 
wrote:

> Hi Lei,
>
> We use 0.8.2. We initially had 0.8.4 but it contains an issue with task
> retry logic so we downgraded to 0.8.2. We are planning to go into
> production with 0.8.2 by next week so can you please advice a better way to
> solve this without upgrading to 0.8.4.
>
> Thanks
> Dimuthu
>
> On Fri, May 31, 2019 at 1:04 PM Lei Xia  wrote:
>
>> Which Helix version do you use?  This may caused by this Zookeeper bug (
>> https://issues.apache.org/jira/browse/ZOOKEEPER-706).  We have upgraded
>> ZkClient in later Helix versions.
>>
>>
>> Lei
>>
>> On Fri, May 31, 2019 at 7:52 AM DImuthu Upeksha <
>> dimuthu.upeks...@gmail.com> wrote:
>>
>>> Hi Folks,
>>>
>>> I'm getting following error in controller log and seems like controller
>>> is
>>> not moving froward after that point
>>>
>>> 2019-05-31 10:47:37,084 [main] INFO  o.a.a.h.i.c.HelixController  -
>>> Starting helix controller
>>> 2019-05-31 10:47:37,089 [main] INFO  o.a.a.c.u.ApplicationSettings  -
>>> Settings loaded from
>>>
>>> file:/home/airavata/staging-deployment/airavata-helix/apache-airavata-controller-0.18-SNAPSHOT/conf/airavata-server.properties
>>> 2019-05-31 10:47:37,091 [Thread-0] INFO  o.a.a.h.i.c.HelixController  -
>>> Connection to helix cluster : AiravataDemoCluster with name :
>>> helixcontroller2
>>> 2019-05-31 10:47:37,092 [Thread-0] INFO  o.a.a.h.i.c.HelixController  -
>>> Zookeeper connection string localhost:2181
>>> 2019-05-31 10:47:42,907 [GenericHelixController-event_process] ERROR
>>> o.a.h.c.GenericHelixController  - Exception while executing
>>> DEFAULTpipeline:
>>> org.apache.helix.controller.pipeline.Pipeline@408d6d26for
>>> cluster .AiravataDemoCluster. Will not continue to next pipeline
>>> org.apache.helix.api.exceptions.HelixMetaDataAccessException: Failed to
>>> get
>>> full list of /AiravataDemoCluster/CONFIGS/PARTICIPANT
>>> at
>>>
>>> org.apache.helix.manager.zk.ZkBaseDataAccessor.getChildren(ZkBaseDataAccessor.java:446)
>>> at
>>>
>>> org.apache.helix.manager.zk.ZKHelixDataAccessor.getChildValues(ZKHelixDataAccessor.java:406)
>>> at
>>>
>>> org.apache.helix.manager.zk.ZKHelixDataAccessor.getChildValuesMap(ZKHelixDataAccessor.java:467)
>>> at
>>>
>>> org.apache.helix.controller.stages.ClusterDataCache.refresh(ClusterDataCache.java:176)
>>> at
>>>
>>> org.apache.helix.controller.stages.ReadClusterDataStage.process(ReadClusterDataStage.java:62)
>>> at org.apache.helix.controller.pipeline.Pipeline.handle(Pipeline.java:63)
>>> at
>>>
>>> org.apache.helix.controller.GenericHelixController.handleEvent(GenericHelixController.java:432)
>>> at
>>>
>>> org.apache.helix.controller.GenericHelixController$ClusterEventProcessor.run(GenericHelixController.java:928)
>>> Caused by: org.apache.helix.api.exceptions.HelixMetaDataAccessException:
>>> Fail to read nodes for
>>> [/AiravataDemoCluster/CONFIGS/PARTICIPANT/helixparticipant]
>>> at
>>>
>>> org.apache.helix.manager.zk.ZkBaseDataAccessor.get(ZkBaseDataAccessor.java:414)
>>> at
>>>
>>> org.apache.helix.manager.zk.ZkBaseDataAccessor.getChildren(ZkBaseDataAccessor.java:479)
>>> at
>>>
>>> org.apache.helix.manager.zk.ZkBaseDataAccessor.getChildren(ZkBaseDataAccessor.java:442)
>>> ... 7 common frames omitted
>>>
>>> In the zookeeper log I can see following warn

Re: Zookeeper connection errors in Helix Controller

[DImuthu Upeksha]

Hi Lei,

We use 0.8.2. We initially had 0.8.4 but it contains an issue with task
retry logic so we downgraded to 0.8.2. We are planning to go into
production with 0.8.2 by next week so can you please advice a better way to
solve this without upgrading to 0.8.4.

Thanks
Dimuthu

On Fri, May 31, 2019 at 1:04 PM Lei Xia  wrote:

> Which Helix version do you use?  This may caused by this Zookeeper bug (
> https://issues.apache.org/jira/browse/ZOOKEEPER-706).  We have upgraded
> ZkClient in later Helix versions.
>
>
> Lei
>
> On Fri, May 31, 2019 at 7:52 AM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com> wrote:
>
>> Hi Folks,
>>
>> I'm getting following error in controller log and seems like controller is
>> not moving froward after that point
>>
>> 2019-05-31 10:47:37,084 [main] INFO  o.a.a.h.i.c.HelixController  -
>> Starting helix controller
>> 2019-05-31 10:47:37,089 [main] INFO  o.a.a.c.u.ApplicationSettings  -
>> Settings loaded from
>>
>> file:/home/airavata/staging-deployment/airavata-helix/apache-airavata-controller-0.18-SNAPSHOT/conf/airavata-server.properties
>> 2019-05-31 10:47:37,091 [Thread-0] INFO  o.a.a.h.i.c.HelixController  -
>> Connection to helix cluster : AiravataDemoCluster with name :
>> helixcontroller2
>> 2019-05-31 10:47:37,092 [Thread-0] INFO  o.a.a.h.i.c.HelixController  -
>> Zookeeper connection string localhost:2181
>> 2019-05-31 10:47:42,907 [GenericHelixController-event_process] ERROR
>> o.a.h.c.GenericHelixController  - Exception while executing
>> DEFAULTpipeline: org.apache.helix.controller.pipeline.Pipeline@408d6d26for
>> cluster .AiravataDemoCluster. Will not continue to next pipeline
>> org.apache.helix.api.exceptions.HelixMetaDataAccessException: Failed to
>> get
>> full list of /AiravataDemoCluster/CONFIGS/PARTICIPANT
>> at
>>
>> org.apache.helix.manager.zk.ZkBaseDataAccessor.getChildren(ZkBaseDataAccessor.java:446)
>> at
>>
>> org.apache.helix.manager.zk.ZKHelixDataAccessor.getChildValues(ZKHelixDataAccessor.java:406)
>> at
>>
>> org.apache.helix.manager.zk.ZKHelixDataAccessor.getChildValuesMap(ZKHelixDataAccessor.java:467)
>> at
>>
>> org.apache.helix.controller.stages.ClusterDataCache.refresh(ClusterDataCache.java:176)
>> at
>>
>> org.apache.helix.controller.stages.ReadClusterDataStage.process(ReadClusterDataStage.java:62)
>> at org.apache.helix.controller.pipeline.Pipeline.handle(Pipeline.java:63)
>> at
>>
>> org.apache.helix.controller.GenericHelixController.handleEvent(GenericHelixController.java:432)
>> at
>>
>> org.apache.helix.controller.GenericHelixController$ClusterEventProcessor.run(GenericHelixController.java:928)
>> Caused by: org.apache.helix.api.exceptions.HelixMetaDataAccessException:
>> Fail to read nodes for
>> [/AiravataDemoCluster/CONFIGS/PARTICIPANT/helixparticipant]
>> at
>>
>> org.apache.helix.manager.zk.ZkBaseDataAccessor.get(ZkBaseDataAccessor.java:414)
>> at
>>
>> org.apache.helix.manager.zk.ZkBaseDataAccessor.getChildren(ZkBaseDataAccessor.java:479)
>> at
>>
>> org.apache.helix.manager.zk.ZkBaseDataAccessor.getChildren(ZkBaseDataAccessor.java:442)
>> ... 7 common frames omitted
>>
>> In the zookeeper log I can see following warning getting printed
>> continuously. What could be the reason for that? I'm using helix 0.8.2 and
>> zookeeper 3.4.8
>>
>> 2019-05-31 10:49:37,621 [myid:] - INFO  [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for
>> client /0:0:0:0:0:0:0:1:59056 which had sessionid 0x16b0e59877f
>> 2019-05-31 10:49:37,773 [myid:] - INFO  [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket
>> connection
>> from /127.0.0.1:57984
>> 2019-05-31 10:49:37,774 [myid:] - INFO  [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@893] - Client attempting to renew
>> session 0x16b0e59877f at /127.0.0.1:57984
>> 2019-05-31 10:49:37,774 [myid:] - INFO  [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@645] - Established session
>> 0x16b0e59877f with negotiated timeout 3 for client /
>> 127.0.0.1:57984
>> 2019-05-31 10:49:37,790 [myid:] - WARN  [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
>> EndOfStreamException: Unable to read additional data from client sessionid
>> 0x16b0e59877f, likely client has closed socket
>> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230)
>> at
>>
>> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> Thanks
>> Dimuthu
>>
>
>
> --
> Lei Xia
>

Re: Zookeeper connection errors in Helix Controller

[DImuthu Upeksha]

Hi Kishore,

Adding -Djute.maxbuffer=49107800 fixed the issue but now I can see a whole
lot of logs printing with following line and participant is executing a
bulk of Tasks once in a while with around 5 minute delay in between.

2019-05-31 12:45:58,804 [GenericHelixController-event_process] WARN
 o.a.h.t.a.AssignableInstance  - AssignableInstance does not have enough
capacity for quotaType: DEFAULT. Task:
Task_TASK_57f87021-bedb-415a-bf1f-ddfd17641f11, quotaType: DEFAULT,
Instance name: helixparticipant3. Current capacity: 40 capacity needed to
schedule: 40

Thanks
Dimuthu

On Fri, May 31, 2019 at 12:25 PM DImuthu Upeksha 
wrote:

> Hi Kishore,
>
> Please find below log and I think the issue is "Packet len4194362 is out
> of range!". Currently we have around 1 unprocessed workflows. Will that
> be the reason for that?
>
> 2019-05-31 12:17:15,221 [Thread-0-EventThread] INFO
>  o.a.h.m.zk.zookeeper.ZkClient  - zookeeper state changed (SyncConnected)
> 2019-05-31 12:17:15,237 [Thread-0-SendThread(localhost:2181)] WARN
>  o.apache.zookeeper.ClientCnxn  - Session 0x16b0e59877f0005 for server
> localhost/0:0:0:0:0:0:0:1:2181, unexpected error, closing socket connection
> and attempting reconnect
> java.io.IOException: Packet len4194362 is out of range!
> at
> org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:112)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)
> 2019-05-31 12:17:15,337 [Thread-0-EventThread] INFO
>  o.a.h.m.zk.zookeeper.ZkClient  - zookeeper state changed (Disconnected)
> 2019-05-31 12:17:15,523 [Thread-0-SendThread(localhost:2181)] INFO
>  o.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> localhost/127.0.0.1:2181
> 2019-05-31 12:17:15,524 [Thread-0-SendThread(localhost:2181)] INFO
>  o.apache.zookeeper.ClientCnxn  - Socket connection established to
> localhost/127.0.0.1:2181, initiating session
> 2019-05-31 12:17:15,525 [Thread-0-SendThread(localhost:2181)] INFO
>  o.apache.zookeeper.ClientCnxn  - Session establishment complete on server
> localhost/127.0.0.1:2181, sessionid = 0x16b0e59877f0005, negotiated
> timeout = 3
> 2019-05-31 12:17:15,526 [Thread-0-EventThread] INFO
>  o.a.h.m.zk.zookeeper.ZkClient  - zookeeper state changed (SyncConnected)
> 2019-05-31 12:17:15,540 [Thread-0-SendThread(localhost:2181)] WARN
>  o.apache.zookeeper.ClientCnxn  - Session 0x16b0e59877f0005 for server
> localhost/127.0.0.1:2181, unexpected error, closing socket connection and
> attempting reconnect
> java.io.IOException: Packet len4194362 is out of range!
> at
> org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:112)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)
> 2019-05-31 12:17:15,641 [Thread-0-EventThread] INFO
>  o.a.h.m.zk.zookeeper.ZkClient  - zookeeper state changed (Disconnected)
>
> Thanks
> Dimuthu
>
> On Fri, May 31, 2019 at 11:55 AM kishore g  wrote:
>
>> can you grep for zookeeper state in controller log.
>>
>> On Fri, May 31, 2019 at 7:52 AM DImuthu Upeksha <
>> dimuthu.upeks...@gmail.com>
>> wrote:
>>
>> > Hi Folks,
>> >
>> > I'm getting following error in controller log and seems like controller
>> is
>> > not moving froward after that point
>> >
>> > 2019-05-31 10:47:37,084 [main] INFO  o.a.a.h.i.c.HelixController  -
>> > Starting helix controller
>> > 2019-05-31 10:47:37,089 [main] INFO  o.a.a.c.u.ApplicationSettings  -
>> > Settings loaded from
>> >
>> >
>> file:/home/airavata/staging-deployment/airavata-helix/apache-airavata-controller-0.18-SNAPSHOT/conf/airavata-server.properties
>> > 2019-05-31 10:47:37,091 [Thread-0] INFO  o.a.a.h.i.c.HelixController  -
>> > Connection to helix cluster : AiravataDemoCluster with name :
>> > helixcontroller2
>> > 2019-05-31 10:47:37,092 [Thread-0] INFO  o.a.a.h.i.c.HelixController  -
>> > Zookeeper connection string localhost:2181
>> > 2019-05-31 10:47:42,907 [GenericHelixController-event_process] ERROR
>> > o.a.h.c.GenericHelixController  - Exception while executing
>> > DEFAULTpipeline:
>> org.apache.helix.controller.pipeline.Pipeline@408d6d26for
>> > cluster .AiravataDemoCluster. Will not continue to next pipeline
>> > org.apache.helix.api.exceptions.HelixMetaDataAccessExce

Re: Zookeeper connection errors in Helix Controller

[DImuthu Upeksha]

Hi Kishore,

Please find below log and I think the issue is "Packet len4194362 is out of
range!". Currently we have around 1 unprocessed workflows. Will that be
the reason for that?

2019-05-31 12:17:15,221 [Thread-0-EventThread] INFO
 o.a.h.m.zk.zookeeper.ZkClient  - zookeeper state changed (SyncConnected)
2019-05-31 12:17:15,237 [Thread-0-SendThread(localhost:2181)] WARN
 o.apache.zookeeper.ClientCnxn  - Session 0x16b0e59877f0005 for server
localhost/0:0:0:0:0:0:0:1:2181, unexpected error, closing socket connection
and attempting reconnect
java.io.IOException: Packet len4194362 is out of range!
at
org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:112)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)
2019-05-31 12:17:15,337 [Thread-0-EventThread] INFO
 o.a.h.m.zk.zookeeper.ZkClient  - zookeeper state changed (Disconnected)
2019-05-31 12:17:15,523 [Thread-0-SendThread(localhost:2181)] INFO
 o.apache.zookeeper.ClientCnxn  - Opening socket connection to server
localhost/127.0.0.1:2181
2019-05-31 12:17:15,524 [Thread-0-SendThread(localhost:2181)] INFO
 o.apache.zookeeper.ClientCnxn  - Socket connection established to
localhost/127.0.0.1:2181, initiating session
2019-05-31 12:17:15,525 [Thread-0-SendThread(localhost:2181)] INFO
 o.apache.zookeeper.ClientCnxn  - Session establishment complete on server
localhost/127.0.0.1:2181, sessionid = 0x16b0e59877f0005, negotiated timeout
= 3
2019-05-31 12:17:15,526 [Thread-0-EventThread] INFO
 o.a.h.m.zk.zookeeper.ZkClient  - zookeeper state changed (SyncConnected)
2019-05-31 12:17:15,540 [Thread-0-SendThread(localhost:2181)] WARN
 o.apache.zookeeper.ClientCnxn  - Session 0x16b0e59877f0005 for server
localhost/127.0.0.1:2181, unexpected error, closing socket connection and
attempting reconnect
java.io.IOException: Packet len4194362 is out of range!
at
org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:112)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)
2019-05-31 12:17:15,641 [Thread-0-EventThread] INFO
 o.a.h.m.zk.zookeeper.ZkClient  - zookeeper state changed (Disconnected)

Thanks
Dimuthu

On Fri, May 31, 2019 at 11:55 AM kishore g  wrote:

> can you grep for zookeeper state in controller log.
>
> On Fri, May 31, 2019 at 7:52 AM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com>
> wrote:
>
> > Hi Folks,
> >
> > I'm getting following error in controller log and seems like controller
> is
> > not moving froward after that point
> >
> > 2019-05-31 10:47:37,084 [main] INFO  o.a.a.h.i.c.HelixController  -
> > Starting helix controller
> > 2019-05-31 10:47:37,089 [main] INFO  o.a.a.c.u.ApplicationSettings  -
> > Settings loaded from
> >
> >
> file:/home/airavata/staging-deployment/airavata-helix/apache-airavata-controller-0.18-SNAPSHOT/conf/airavata-server.properties
> > 2019-05-31 10:47:37,091 [Thread-0] INFO  o.a.a.h.i.c.HelixController  -
> > Connection to helix cluster : AiravataDemoCluster with name :
> > helixcontroller2
> > 2019-05-31 10:47:37,092 [Thread-0] INFO  o.a.a.h.i.c.HelixController  -
> > Zookeeper connection string localhost:2181
> > 2019-05-31 10:47:42,907 [GenericHelixController-event_process] ERROR
> > o.a.h.c.GenericHelixController  - Exception while executing
> > DEFAULTpipeline:
> org.apache.helix.controller.pipeline.Pipeline@408d6d26for
> > cluster .AiravataDemoCluster. Will not continue to next pipeline
> > org.apache.helix.api.exceptions.HelixMetaDataAccessException: Failed to
> get
> > full list of /AiravataDemoCluster/CONFIGS/PARTICIPANT
> > at
> >
> >
> org.apache.helix.manager.zk.ZkBaseDataAccessor.getChildren(ZkBaseDataAccessor.java:446)
> > at
> >
> >
> org.apache.helix.manager.zk.ZKHelixDataAccessor.getChildValues(ZKHelixDataAccessor.java:406)
> > at
> >
> >
> org.apache.helix.manager.zk.ZKHelixDataAccessor.getChildValuesMap(ZKHelixDataAccessor.java:467)
> > at
> >
> >
> org.apache.helix.controller.stages.ClusterDataCache.refresh(ClusterDataCache.java:176)
> > at
> >
> >
> org.apache.helix.controller.stages.ReadClusterDataStage.process(ReadClusterDataStage.java:62)
> > at org.apache.helix.controller.pipeline.Pipeline.handle(Pipeline.java:63)
> > at
> >
> >
> org.apache.helix.controller.GenericHelixController.handleEvent(GenericHelixController.java:432)
> > at
> >
> >
> org.apache.hel

Zookeeper connection errors in Helix Controller

[DImuthu Upeksha]

Hi Folks,

I'm getting following error in controller log and seems like controller is
not moving froward after that point

2019-05-31 10:47:37,084 [main] INFO  o.a.a.h.i.c.HelixController  -
Starting helix controller
2019-05-31 10:47:37,089 [main] INFO  o.a.a.c.u.ApplicationSettings  -
Settings loaded from
file:/home/airavata/staging-deployment/airavata-helix/apache-airavata-controller-0.18-SNAPSHOT/conf/airavata-server.properties
2019-05-31 10:47:37,091 [Thread-0] INFO  o.a.a.h.i.c.HelixController  -
Connection to helix cluster : AiravataDemoCluster with name :
helixcontroller2
2019-05-31 10:47:37,092 [Thread-0] INFO  o.a.a.h.i.c.HelixController  -
Zookeeper connection string localhost:2181
2019-05-31 10:47:42,907 [GenericHelixController-event_process] ERROR
o.a.h.c.GenericHelixController  - Exception while executing
DEFAULTpipeline: org.apache.helix.controller.pipeline.Pipeline@408d6d26for
cluster .AiravataDemoCluster. Will not continue to next pipeline
org.apache.helix.api.exceptions.HelixMetaDataAccessException: Failed to get
full list of /AiravataDemoCluster/CONFIGS/PARTICIPANT
at
org.apache.helix.manager.zk.ZkBaseDataAccessor.getChildren(ZkBaseDataAccessor.java:446)
at
org.apache.helix.manager.zk.ZKHelixDataAccessor.getChildValues(ZKHelixDataAccessor.java:406)
at
org.apache.helix.manager.zk.ZKHelixDataAccessor.getChildValuesMap(ZKHelixDataAccessor.java:467)
at
org.apache.helix.controller.stages.ClusterDataCache.refresh(ClusterDataCache.java:176)
at
org.apache.helix.controller.stages.ReadClusterDataStage.process(ReadClusterDataStage.java:62)
at org.apache.helix.controller.pipeline.Pipeline.handle(Pipeline.java:63)
at
org.apache.helix.controller.GenericHelixController.handleEvent(GenericHelixController.java:432)
at
org.apache.helix.controller.GenericHelixController$ClusterEventProcessor.run(GenericHelixController.java:928)
Caused by: org.apache.helix.api.exceptions.HelixMetaDataAccessException:
Fail to read nodes for
[/AiravataDemoCluster/CONFIGS/PARTICIPANT/helixparticipant]
at
org.apache.helix.manager.zk.ZkBaseDataAccessor.get(ZkBaseDataAccessor.java:414)
at
org.apache.helix.manager.zk.ZkBaseDataAccessor.getChildren(ZkBaseDataAccessor.java:479)
at
org.apache.helix.manager.zk.ZkBaseDataAccessor.getChildren(ZkBaseDataAccessor.java:442)
... 7 common frames omitted

In the zookeeper log I can see following warning getting printed
continuously. What could be the reason for that? I'm using helix 0.8.2 and
zookeeper 3.4.8

2019-05-31 10:49:37,621 [myid:] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for
client /0:0:0:0:0:0:0:1:59056 which had sessionid 0x16b0e59877f
2019-05-31 10:49:37,773 [myid:] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection
from /127.0.0.1:57984
2019-05-31 10:49:37,774 [myid:] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@893] - Client attempting to renew
session 0x16b0e59877f at /127.0.0.1:57984
2019-05-31 10:49:37,774 [myid:] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@645] - Established session
0x16b0e59877f with negotiated timeout 3 for client /127.0.0.1:57984
2019-05-31 10:49:37,790 [myid:] - WARN  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid
0x16b0e59877f, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
at java.lang.Thread.run(Thread.java:748)

Thanks
Dimuthu

Cleaning up unused code

[DImuthu Upeksha]

Hi Folks,

I'm working on $subject and reducing the size of distributions with minimal
dependencies. I will keep adding the classes that I'm going to remove to
this thread and please let me know whether  those are required.

https://github.com/apache/airavata/blob/master/airavata-services/services-security/src/main/java/org/apache/airavata/service/security/DefaultAiravataSecurityManager.java

https://github.com/apache/airavata/blob/master/airavata-services/services-security/src/main/java/org/apache/airavata/service/security/oauth/DefaultOAuthClient.java

https://github.com/apache/airavata/blob/master/airavata-services/services-security/src/main/java/org/apache/airavata/service/security/Main.java


https://github.com/apache/airavata/blob/master/airavata-services/services-security/src/main/java/org/apache/airavata/service/security/xacml/DefaultPAPClient.java

https://github.com/apache/airavata/blob/master/airavata-services/services-security/src/main/java/org/apache/airavata/service/security/xacml/DefaultXACMLPEP.java

Thanks
Dimuthu

Re: Submitted the project draft - Metascheduler for Airavata

[DImuthu Upeksha]

Hi Bibrak,

Please go through [1] this mail thread and capture current use cases for
new scheduler. You can summarize them in the proposal.

[1] https://www.mail-archive.com/dev@airavata.apache.org/msg12180.html

Thanks
Dimuthu

On Mon, Apr 8, 2019 at 2:44 PM Bibrak Qamar  wrote:

> Hi all,
>
> I have submitted by proposal draft at the GSoC site I am also sharing the
> link here too:
> https://docs.google.com/document/d/1tK4rjKCycwDawJBuEgOEDJ9oAQNLS6baEREbSoxFIlQ/edit?usp=sharing
>
> Could you please have a look at it and help me improve it by providing
> your comments and suggestions?
>
> Thanks,
> Bibrak Qamar
>
>

Re: ZkHelixManager disconnection hangs

[DImuthu Upeksha]

Hi Jiajun, Kishore and others,

Thanks for looking into this. We used 0.8.1 in that setup and now we
upgraded it to 0.8.2. However this was a frequently occurring issue causing
controller to manually kill and restart and we expect it to happen again if
it isn't fixed in 0.8.2. Main reason for that is we are relying on a single
Zookeper node across a fairly unreliable network for our staging
environment. We will let you know if we see that issue again. In the
meantime, I will try to reproduce it in 0.8.2 in my local helix deployment.

Thanks
Dimuthu

On Tue, Apr 2, 2019 at 1:58 AM ericwang1985  wrote:

> Could you please confirm the Helix version that is used, Dimuthu?
> The thing is that we have fixed several potential ZkHelixManager
> concurrency issues in 0.8.2. Basically, that was a race condition in which
> the disconnect method could get a disconnected non-null zkclient. In this
> case, reset handler will never finish.
>
> Please let us know if you are already using 0.8.2 or a later version. That
> probably means we have a new bug to fix.
>
> Cheers,
> -Jiajun
>
> On Apr 1, 2019, at 13:15, kishore g  wrote:
>
> This is a good catch. @Wang Jiajun  the stack
> trace is good enough to fix this right. We just have to look at all the
> paths we can get into this method and make sure resetHandler is thread safe
> and validates the state of the zkConnection and handlers.
>
> On Mon, Apr 1, 2019 at 12:41 PM Wang Jiajun 
> wrote:
>
>> Hi Dimuthu,
>>
>> Did you stop the controller when the connection is flapping or when it is
>> normal?
>> Could you please list all the steps that you have done in order?
>>
>> Best Regards,
>> Jiajun
>>
>>
>> On Sat, Mar 30, 2019 at 5:54 AM DImuthu Upeksha <
>> dimuthu.upeks...@gmail.com>
>> wrote:
>>
>> > Hi Folks,
>> >
>> > In helix controller, we have seen below log line and by looking at the
>> > code, I understood that it is due to ZkHelixManager is failing to
>> connect
>> > to zookeeper for 5 times. So I tried to stop the controller and in the
>> stop
>> > logic, we have a call to ZkHelixManager.disconnect() method and it
>> hangs. I
>> > got a thread dump and you can see where it is waiting. Can you please
>> > advice as better approach to solve this?
>> >
>> > I noticed that ZkHelixManager disconnects [1] it self when a flapping is
>> > detected. Is calling disconnect() twice the reason for that?
>> >
>> > 2019-03-29 15:19:56,832 [
>> > ZkClient-EventThread-14-api.staging.scigap.org:2181
>> <http://zkclient-eventthread-14-api.staging.scigap.org:2181/>]
>> > ERROR o.a.h.m.zk.ZKHelixManager  - instanceName: helixcontroller is
>> > flapping. disconnect it.  maxDisconnectThreshold: 5 disconnects in
>> > 30ms.
>> >
>> > Thread-5 - priority:5 - threadId:0x7f5c740023f0 - nativeId:0x63f1 -
>> > nativeId (decimal):25585 - state:BLOCKED
>> > stackTrace:
>> > java.lang.Thread.State: BLOCKED (on object monitor)
>> > at
>> >
>> >
>> org.apache.helix.manager.zk.ZKHelixManager.resetHandlers(ZKHelixManager.java:903)
>> > - waiting to lock <0x0006c7e08110> (a
>> > org.apache.helix.manager.zk.ZKHelixManager)
>> > at
>> >
>> >
>> org.apache.helix.manager.zk.ZKHelixManager.disconnect(ZKHelixManager.java:693)
>> > at
>> >
>> >
>> org.apache.airavata.helix.impl.controller.HelixController.disconnect(HelixController.java:103)
>> > at
>> >
>> >
>> org.apache.airavata.helix.impl.controller.HelixController$$Lambda$2/846492085.run(Unknown
>> > Source)
>> > at java.lang.Thread.run(Thread.java:748)
>> > Locked ownable synchronizers:
>> > - None
>> >
>> > [1]
>> >
>> >
>> https://github.com/apache/helix/blob/helix-0.8.2/helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixManager.java#L991
>> > Thanks
>> > Dimuthu
>> >
>>
>
>

Re: Sporadic delays in task execution

[DImuthu Upeksha]

Hi Lee,

Thanks for letting us know. We would happy to try out the latest version.
Can you please point me to such known issues (JIRA or Github issues) in
latest version and then we can decide whether those issue might affect to
our use case or not.

Thanks
Dimuthu

On Fri, Mar 22, 2019 at 4:29 PM Hunter Lee  wrote:

> Let me add a caveat to my previous email. Although it comes with
> scalability improvements, there are currently a few known issues with the
> latest version. We'd encourage you to check back to make sure your current
> usage isn't affected.
>
> Hunter
>
> On Fri, Mar 22, 2019 at 12:35 PM Hunter Lee  wrote:
>
> > No problem. If you have further questions, let us know what kind of load
> > you're putting on Helix as well. The newest version of Helix contains
> Task
> > Framework 2.0, and has greater scalability in scheduling tasks, so you
> > might want to consider using the newest version as well.
> >
> > Hunter
> >
> > On Fri, Mar 22, 2019 at 8:59 AM DImuthu Upeksha <
> > dimuthu.upeks...@gmail.com> wrote:
> >
> >> Hi Lee,
> >>
> >> Thanks for the trick. I didn't know that we can poke the controller like
> >> that :) However now we can see that tasks are moving smoothly in our
> >> staging setup. This behavior can be seen from time to time and get
> >> resolved
> >> automatically in few hours. I can't find a particular pattern however my
> >> best guess is that this happens when the load is high. I will put some
> >> load
> >> on testing setup and see if I can reproduce this issue and try your
> >> instructions then get back to you
> >>
> >> Thanks
> >> Dimuthu
> >>
> >> On Thu, Mar 21, 2019 at 5:27 PM Hunter Lee  wrote:
> >>
> >> > Hi Dimuthu,
> >> >
> >> > What Junkai meant by touching the IdealState is this:
> >> >
> >> > 1) use Zooinspector to log into ZK
> >> > 2) Locate the IDEALSTATES/ path
> >> > 3) grab any ZNode under that path and try to modify (just add a
> >> > whitespace) and save
> >> > 4) This will trigger a ZK callback which should tell Helix Controller
> to
> >> > rebalance/schedule things
> >> >
> >> > On Thu, Mar 21, 2019 at 11:30 AM DImuthu Upeksha <
> >> > dimuthu.upeks...@gmail.com> wrote:
> >> >
> >> >> Hi Junkai,
> >> >>
> >> >> What do you mean by touching ideal state to trigger an event? I
> didn't
> >> >> quite get what you said. Is that like creating some path in
> zookeeper?
> >> >> Workflows are eventually scheduled but the problem is, it is very
> slow
> >> due
> >> >> to that 30s freeze.
> >> >>
> >> >> Thanks
> >> >> Dimuthu
> >> >>
> >> >> On Thu, Mar 21, 2019 at 2:26 PM Xue Junkai 
> >> wrote:
> >> >>
> >> >> > Can you try one thing? Touch the ideal state to trigger an event.
> If
> >> >> > workflows are not scheduled, it should scheduling has problem.
> >> >> >
> >> >> > Best,
> >> >> >
> >> >> > Junkai
> >> >> >
> >> >> > On Wed, Mar 20, 2019 at 10:31 PM DImuthu Upeksha <
> >> >> > dimuthu.upeks...@gmail.com> wrote:
> >> >> >
> >> >> >> Hi Junkai,
> >> >> >>
> >> >> >> We are using 0.8.1
> >> >> >>
> >> >> >> Dimuthu
> >> >> >>
> >> >> >> On Thu, Mar 21, 2019 at 12:14 AM Xue Junkai  >
> >> >> wrote:
> >> >> >>
> >> >> >> > Hi Dimuthu,
> >> >> >> >
> >> >> >> > What's the version of Helix you are using?
> >> >> >> >
> >> >> >> > Best,
> >> >> >> >
> >> >> >> > Junkai
> >> >> >> >
> >> >> >> > On Wed, Mar 20, 2019 at 8:54 PM DImuthu Upeksha <
> >> >> >> > dimuthu.upeks...@gmail.com>
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> > > Hi Helix Dev,
> >> >> >> > >
> >> >> >> > > We are again seeing this delay in task execution. Please have
> a
> >> >> look
> >> >> 

Re: Sporadic delays in task execution

[DImuthu Upeksha]

Hi Lee,

Thanks for the trick. I didn't know that we can poke the controller like
that :) However now we can see that tasks are moving smoothly in our
staging setup. This behavior can be seen from time to time and get resolved
automatically in few hours. I can't find a particular pattern however my
best guess is that this happens when the load is high. I will put some load
on testing setup and see if I can reproduce this issue and try your
instructions then get back to you

Thanks
Dimuthu

On Thu, Mar 21, 2019 at 5:27 PM Hunter Lee  wrote:

> Hi Dimuthu,
>
> What Junkai meant by touching the IdealState is this:
>
> 1) use Zooinspector to log into ZK
> 2) Locate the IDEALSTATES/ path
> 3) grab any ZNode under that path and try to modify (just add a
> whitespace) and save
> 4) This will trigger a ZK callback which should tell Helix Controller to
> rebalance/schedule things
>
> On Thu, Mar 21, 2019 at 11:30 AM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com> wrote:
>
>> Hi Junkai,
>>
>> What do you mean by touching ideal state to trigger an event? I didn't
>> quite get what you said. Is that like creating some path in zookeeper?
>> Workflows are eventually scheduled but the problem is, it is very slow due
>> to that 30s freeze.
>>
>> Thanks
>> Dimuthu
>>
>> On Thu, Mar 21, 2019 at 2:26 PM Xue Junkai  wrote:
>>
>> > Can you try one thing? Touch the ideal state to trigger an event. If
>> > workflows are not scheduled, it should scheduling has problem.
>> >
>> > Best,
>> >
>> > Junkai
>> >
>> > On Wed, Mar 20, 2019 at 10:31 PM DImuthu Upeksha <
>> > dimuthu.upeks...@gmail.com> wrote:
>> >
>> >> Hi Junkai,
>> >>
>> >> We are using 0.8.1
>> >>
>> >> Dimuthu
>> >>
>> >> On Thu, Mar 21, 2019 at 12:14 AM Xue Junkai 
>> wrote:
>> >>
>> >> > Hi Dimuthu,
>> >> >
>> >> > What's the version of Helix you are using?
>> >> >
>> >> > Best,
>> >> >
>> >> > Junkai
>> >> >
>> >> > On Wed, Mar 20, 2019 at 8:54 PM DImuthu Upeksha <
>> >> > dimuthu.upeks...@gmail.com>
>> >> > wrote:
>> >> >
>> >> > > Hi Helix Dev,
>> >> > >
>> >> > > We are again seeing this delay in task execution. Please have a
>> look
>> >> at
>> >> > the
>> >> > > screencast [1] of logs printed in participant (top shell) and
>> >> controller
>> >> > > (bottom shell). When I record this, there were about 90 - 100
>> >> workflows
>> >> > > pending to be executed. As you can see some tasks were suddenly
>> >> executed
>> >> > > and then participant freezed for about 30 seconds before executing
>> >> next
>> >> > set
>> >> > > of tasks. I can see some WARN logs on controller log. I feel like
>> >> this 30
>> >> > > second delay is some sort of a pattern. What do you think as the
>> >> reason
>> >> > for
>> >> > > this? I can provide you more information by turning on verbose
>> logs on
>> >> > > controller if you want.
>> >> > >
>> >> > > [1] https://youtu.be/3EUdSxnIxVw
>> >> > >
>> >> > > Thanks
>> >> > > Dimuthu
>> >> > >
>> >> > > On Thu, Oct 4, 2018 at 4:46 PM DImuthu Upeksha <
>> >> > dimuthu.upeks...@gmail.com
>> >> > > >
>> >> > > wrote:
>> >> > >
>> >> > > > Hi Junkai,
>> >> > > >
>> >> > > > I'm CCing Airavata dev list as this is directly related to the
>> >> project.
>> >> > > >
>> >> > > > I just went through the zookeeper path like /> >> > Name>/EXTERNALVIEW,
>> >> > > > //CONFIGS/RESOURCE as I have noticed that helix
>> >> > controller
>> >> > > is
>> >> > > > periodically monitoring for the children of those paths even
>> though
>> >> all
>> >> > > the
>> >> > > > Workflows have moved into a saturated state like COMPLETED and
>> >> STOPPED.
>> >> > > In
>> >> > > > our case, we have a lot of completed workflows pil

Re: Sporadic delays in task execution

[DImuthu Upeksha]

Hi Junkai,

What do you mean by touching ideal state to trigger an event? I didn't
quite get what you said. Is that like creating some path in zookeeper?
Workflows are eventually scheduled but the problem is, it is very slow due
to that 30s freeze.

Thanks
Dimuthu

On Thu, Mar 21, 2019 at 2:26 PM Xue Junkai  wrote:

> Can you try one thing? Touch the ideal state to trigger an event. If
> workflows are not scheduled, it should scheduling has problem.
>
> Best,
>
> Junkai
>
> On Wed, Mar 20, 2019 at 10:31 PM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com> wrote:
>
>> Hi Junkai,
>>
>> We are using 0.8.1
>>
>> Dimuthu
>>
>> On Thu, Mar 21, 2019 at 12:14 AM Xue Junkai  wrote:
>>
>> > Hi Dimuthu,
>> >
>> > What's the version of Helix you are using?
>> >
>> > Best,
>> >
>> > Junkai
>> >
>> > On Wed, Mar 20, 2019 at 8:54 PM DImuthu Upeksha <
>> > dimuthu.upeks...@gmail.com>
>> > wrote:
>> >
>> > > Hi Helix Dev,
>> > >
>> > > We are again seeing this delay in task execution. Please have a look
>> at
>> > the
>> > > screencast [1] of logs printed in participant (top shell) and
>> controller
>> > > (bottom shell). When I record this, there were about 90 - 100
>> workflows
>> > > pending to be executed. As you can see some tasks were suddenly
>> executed
>> > > and then participant freezed for about 30 seconds before executing
>> next
>> > set
>> > > of tasks. I can see some WARN logs on controller log. I feel like
>> this 30
>> > > second delay is some sort of a pattern. What do you think as the
>> reason
>> > for
>> > > this? I can provide you more information by turning on verbose logs on
>> > > controller if you want.
>> > >
>> > > [1] https://youtu.be/3EUdSxnIxVw
>> > >
>> > > Thanks
>> > > Dimuthu
>> > >
>> > > On Thu, Oct 4, 2018 at 4:46 PM DImuthu Upeksha <
>> > dimuthu.upeks...@gmail.com
>> > > >
>> > > wrote:
>> > >
>> > > > Hi Junkai,
>> > > >
>> > > > I'm CCing Airavata dev list as this is directly related to the
>> project.
>> > > >
>> > > > I just went through the zookeeper path like /> > Name>/EXTERNALVIEW,
>> > > > //CONFIGS/RESOURCE as I have noticed that helix
>> > controller
>> > > is
>> > > > periodically monitoring for the children of those paths even though
>> all
>> > > the
>> > > > Workflows have moved into a saturated state like COMPLETED and
>> STOPPED.
>> > > In
>> > > > our case, we have a lot of completed workflows piled up in those
>> > paths. I
>> > > > believe that helix is clearing up those resources after some TTL.
>> What
>> > I
>> > > > did was writing an external spectator [1] that continuously monitors
>> > for
>> > > > saturated workflows and clearing up resources before controller does
>> > that
>> > > > after a TTL. After that, we didn't see such delays in workflow
>> > execution
>> > > > and everything seems to be running smoothly. However we are
>> > continuously
>> > > > monitoring our deployments for any form of adverse effect
>> introduced by
>> > > > that improvement.
>> > > >
>> > > > Please let us know if we are doing something wrong in this
>> improvement
>> > or
>> > > > is there any better way to achieve this directly through helix task
>> > > > framework.
>> > > >
>> > > > [1]
>> > > >
>> > >
>> >
>> https://github.com/apache/airavata/blob/staging/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/controller/WorkflowCleanupAgent.java
>> > > >
>> > > > Thanks
>> > > > Dimuthu
>> > > >
>> > > > On Tue, Oct 2, 2018 at 1:12 PM Xue Junkai 
>> > wrote:
>> > > >
>> > > >> Could you please check the log of how long for each pipeline stage
>> > > takes?
>> > > >>
>> > > >> Also, did you set expiry for workflows? Are they piled up for long
>> > time?
>> > > >> How long for each workflow completes?
>&g

Re: [VOTE] Apache Airavata release 0.17 - RC1

[DImuthu Upeksha]

+1

On Thu, Mar 21, 2019 at 1:07 AM Suresh Marru  wrote:

> Apache Airavata PMC is pleased to call for a vote on the following Apache 
> Airavata 0.17 release candidate artifacts:
>
> Detailed change log/release 
> notes:https://github.com/apache/airavata/blob/airavata-0.17/RELEASE_NOTES
>
> All Release 
> Artifacts:https://dist.apache.org/repos/dist/dev/airavata/0.17/RC1/
>
> PGP release keys (signed using 617DDBAD):
>
> https://dist.apache.org/repos/dist/dev/airavata/KEYS
>
> Specific URL’s:
>
> GIT source tag (305cccab1d8eab8aff28e0fa06ebe9f01ffdde2e):
>
> https://github.com/apache/airavata/tree/airavata-0.17
>
> Source 
> release:https://dist.apache.org/repos/dist/dev/airavata/0.17/RC1/airavata-0.17-source-release.zip
>
> Binary 
> Artifacts:https://dist.apache.org/repos/dist/dev/airavata/0.17/RC1/apache-airavata-server-0.17-bin.tar.gz
>  
> https://dist.apache.org/repos/dist/dev/airavata/0.17/RC1/apache-airavata-server-0.17-bin.zip
>
> Maven staging 
> repo:https://repository.apache.org/content/repositories/orgapacheairavata-1008/
>
> Please verify the artifacts and vote. The vote will be open for 72 hours.
>
> [ ] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)
>
>
>

Re: [DISCUSS] Apache Airavata release 0.17 - RC1

[DImuthu Upeksha]

Sounds good Suresh

Dimuthu

On Thu, Mar 21, 2019 at 9:18 AM Suresh Marru  wrote:

> Hi Dimuthu,
>
> Lets see if there are any other issues, if not, unless you think this is
> blocker, my suggestion is to go ahead with this release since we will
> follow up with a 0.18 please immediately after. We can put a note on
> downloads page.
>
> Cheers,
> Suresh
>
> On Mar 21, 2019, at 1:54 AM, DImuthu Upeksha 
> wrote:
>
> Hi Suresh,
>
> Build succeeded in JDK 1.8 with Maven 3.5.2 on Mac OS. INSTALL file in the
> binary distribution looks little bit out dated.
>
> Thanks
> Dimuthu
>
> On Thu, Mar 21, 2019 at 1:07 AM Suresh Marru  wrote:
>
>> Discussion thread for vote on Apache Airavata 0.17 release candidate.
>>
>> If you have any questions or feedback or to post results of validating the 
>> release, please reply to this thread. Once you verify the release, please 
>> post your vote to the VOTE thread.
>>
>> For reference, the Apache release guide  - 
>> http://www.apache.org/dev/release.html
>>
>> Some tips to validate the release before you vote:
>>
>> * Download the binary version and run the 5 minute or 10 minute tutorial as 
>> described in README and website.
>> * Download the source files from compressed files and release tag and build 
>> (which includes tests).
>> * Verify the distribution for the required LICENSE and NOTICE files
>> * Verify if all the staged files are signed and the signature is verifiable.
>> * Verify if the signing key in the project's KEYS file is hosted on a public 
>> server
>>
>> Thanks for your time in validating the release and voting,
>> Suresh
>> (On Behalf of Airavata PMC)
>>
>>
>>
>

Re: [DISCUSS] Apache Airavata release 0.17 - RC1

[DImuthu Upeksha]

Hi Suresh,

Build succeeded in JDK 1.8 with Maven 3.5.2 on Mac OS. INSTALL file in the
binary distribution looks little bit out dated.

Thanks
Dimuthu

On Thu, Mar 21, 2019 at 1:07 AM Suresh Marru  wrote:

> Discussion thread for vote on Apache Airavata 0.17 release candidate.
>
> If you have any questions or feedback or to post results of validating the 
> release, please reply to this thread. Once you verify the release, please 
> post your vote to the VOTE thread.
>
> For reference, the Apache release guide  - 
> http://www.apache.org/dev/release.html
>
> Some tips to validate the release before you vote:
>
> * Download the binary version and run the 5 minute or 10 minute tutorial as 
> described in README and website.
> * Download the source files from compressed files and release tag and build 
> (which includes tests).
> * Verify the distribution for the required LICENSE and NOTICE files
> * Verify if all the staged files are signed and the signature is verifiable.
> * Verify if the signing key in the project's KEYS file is hosted on a public 
> server
>
> Thanks for your time in validating the release and voting,
> Suresh
> (On Behalf of Airavata PMC)
>
>
>

Re: Sporadic delays in task execution

[DImuthu Upeksha]

Hi Junkai,

We are using 0.8.1

Dimuthu

On Thu, Mar 21, 2019 at 12:14 AM Xue Junkai  wrote:

> Hi Dimuthu,
>
> What's the version of Helix you are using?
>
> Best,
>
> Junkai
>
> On Wed, Mar 20, 2019 at 8:54 PM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com>
> wrote:
>
> > Hi Helix Dev,
> >
> > We are again seeing this delay in task execution. Please have a look at
> the
> > screencast [1] of logs printed in participant (top shell) and controller
> > (bottom shell). When I record this, there were about 90 - 100 workflows
> > pending to be executed. As you can see some tasks were suddenly executed
> > and then participant freezed for about 30 seconds before executing next
> set
> > of tasks. I can see some WARN logs on controller log. I feel like this 30
> > second delay is some sort of a pattern. What do you think as the reason
> for
> > this? I can provide you more information by turning on verbose logs on
> > controller if you want.
> >
> > [1] https://youtu.be/3EUdSxnIxVw
> >
> > Thanks
> > Dimuthu
> >
> > On Thu, Oct 4, 2018 at 4:46 PM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com
> > >
> > wrote:
> >
> > > Hi Junkai,
> > >
> > > I'm CCing Airavata dev list as this is directly related to the project.
> > >
> > > I just went through the zookeeper path like / Name>/EXTERNALVIEW,
> > > //CONFIGS/RESOURCE as I have noticed that helix
> controller
> > is
> > > periodically monitoring for the children of those paths even though all
> > the
> > > Workflows have moved into a saturated state like COMPLETED and STOPPED.
> > In
> > > our case, we have a lot of completed workflows piled up in those
> paths. I
> > > believe that helix is clearing up those resources after some TTL. What
> I
> > > did was writing an external spectator [1] that continuously monitors
> for
> > > saturated workflows and clearing up resources before controller does
> that
> > > after a TTL. After that, we didn't see such delays in workflow
> execution
> > > and everything seems to be running smoothly. However we are
> continuously
> > > monitoring our deployments for any form of adverse effect introduced by
> > > that improvement.
> > >
> > > Please let us know if we are doing something wrong in this improvement
> or
> > > is there any better way to achieve this directly through helix task
> > > framework.
> > >
> > > [1]
> > >
> >
> https://github.com/apache/airavata/blob/staging/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/controller/WorkflowCleanupAgent.java
> > >
> > > Thanks
> > > Dimuthu
> > >
> > > On Tue, Oct 2, 2018 at 1:12 PM Xue Junkai 
> wrote:
> > >
> > >> Could you please check the log of how long for each pipeline stage
> > takes?
> > >>
> > >> Also, did you set expiry for workflows? Are they piled up for long
> time?
> > >> How long for each workflow completes?
> > >>
> > >> best,
> > >>
> > >> Junkai
> > >>
> > >> On Wed, Sep 26, 2018 at 8:52 AM DImuthu Upeksha <
> > >> dimuthu.upeks...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Junkai,
> > >> >
> > >> > Average load is like 10 - 20 workflows per minutes. In some cases
> it's
> > >> less
> > >> > than that However based on the observations, I feel like it does not
> > >> depend
> > >> > on the load and it is sporadic. Is there a particular log lines
> that I
> > >> can
> > >> > filter in controller and participant to capture the timeline of
> > >> workflow so
> > >> > that I can figure out which which component is malfunctioning? We
> use
> > >> helix
> > >> > v 0.8.1.
> > >> >
> > >> > Thanks
> > >> > Dimuthu
> > >> >
> > >> > On Tue, Sep 25, 2018 at 5:19 PM Xue Junkai 
> > >> wrote:
> > >> >
> > >> > > Hi Dimuthu,
> > >> > >
> > >> > > At which rate, you are keep submitting workflows? Usually,
> Workflow
> > >> > > scheduling is very fast. And which version of Helix you are using?
> > >> > >
> > >> > > Best,
> > >> > >