Re: [ANNOUNCE] Welcome Lahiru Jayathilake as Airavata PMC member/committer

2024-04-08 Thread Dimuthu Upeksha
Welcome, Lahiru. Looking forward for your continuing contributions Thanks Dimuthu On Mon, Apr 8, 2024 at 5:43 PM Suresh Marru wrote: > Hi All, > > The Project Management Committee (PMC) for Apache Airavata has asked Lahiru > Jayathilake to become a committer and PMC member based on his

Re: Airavata-MFT | Local transport implementation

2023-03-21 Thread Dimuthu Upeksha
Hi Praneeth, SCP transport is now available in MFT [1] and working fine with S3. You can update your load testing scripts to measure the performance. [1] https://github.com/apache/airavata-mft/commit/a61fa4a34129cf0e7e807ee23461237a8a72af22 Thanks Dimuthu On Mon, Mar 20, 2023 at 3:43 PM

Re: Cybershuttle Replica Catalog API

2023-02-22 Thread DImuthu Upeksha
Hi Jayan, This looks great and the APIs are clear enough to understand. Specifically, automatic documentation generation is a very nice feature. I would like to suggest a few modifications to the replica data models though. I understand that you got an idea about the replica data models through

Re: Regarding Pre and Post job commands in Airavata

2023-01-05 Thread DImuthu Upeksha
Hi Dinuka, Sorry for the late reply. It is great to explore options to integrate Dag capabilities into the job submitter. We already have some form of HTCondor support in Airavata. Can you summarize the difference between what we already have for HTCondor and this Dag feature? I am specifically

Re: Custos baremetal deployment - need domain names for jetstream2 VMs

2022-07-20 Thread DImuthu Upeksha
Hi Abhinav, If you have created VMs through Exosphere, there is an inbuilt domain name mapping mechanism called UAP[1]. You can directly use that and it comes with certificates configured. Let me know if you do not understand the naming convention [1]

Re: DockerSpawnner for Jupyterhub

2022-07-01 Thread DImuthu Upeksha
Hi Himanshu, Use the standard batch spawner provided by jupyterehub. Once you have it running in a SLURM cluster, you need to change the slurm script that spins up notebook processes to spinup notebook containers instead. The container image should be configurable. [1]

Re: Regarding seagrid desktop client molecule viewer inetgration

2022-07-01 Thread DImuthu Upeksha
Hi Aishwarya, This is a very solid improvement. Is there a way we can try these out? Do you have a working fork? Thanks Dimuthu On Fri, Jul 1, 2022 at 12:59 PM Pamidighantam, Sudhakar wrote: > Aishwarya: > > > > Looks good. Keep this and then look at JS molecular Editors or third party >

Re: Airavata Django Portal

2022-06-30 Thread DImuthu Upeksha
etting the same error. >> Herewith I've attached the logs >> >> On Mon, Jun 27, 2022 at 11:16 PM DImuthu Upeksha < >> dimuthu.upeks...@gmail.com> wrote: >> >>> Seems like your API server is not running properly. There are >>> unoptimized JPA cl

Re: Abhinav's GSoC progress June 27

2022-06-27 Thread DImuthu Upeksha
Hi Abhinav, Thanks for the update. It would be great if there is a working PR for ansible changes Thanks Dimuthu On Mon, Jun 27, 2022 at 10:06 PM Abhinav Sinha wrote: > Hi Dev, > > > > Over the last couple of weeks – I was shadowing Isuru’s work on Custos > bare metal deployment – especially

Re: [jupyter-platform-dashboard] No database migration script available

2022-06-27 Thread DImuthu Upeksha
Hi Dinuka, Please use the new code base at [3]. Start the API by running it inside the IDE. Right-click on Application.java and run should work. UI can be started by running npm start. [3] https://github.com/SciGaP/iPython-Kerner-Changes/tree/main/jupyter-platform Thanks Dimuthu On Mon, Jun

Re: Airavata Django Portal

2022-06-27 Thread DImuthu Upeksha
; When I telnet to airavata.host:8443 I get the following. And I've attached > the logs as well. Hope these help. Thanks in advance. > > [image: Screenshot 2022-06-27 at 22.47.59.png] > > On Sun, Jun 26, 2022 at 9:52 PM DImuthu Upeksha < > dimuthu.upeks...@gmail.com> wrote: > >

Re: [apache/airavata-sandbox] Jupyter Platform Dashboard (PR #59)

2022-06-26 Thread DImuthu Upeksha
Hi Himanshu Thanks for sending the PR. I merged your code to sandbox and it looks good. I will gradually borrow react code into the original scigap ipython repository [1]. I have written a java-based API for the backend instead of flask because it is easy to implement the rest of the

Re: Airavata Django Portal

2022-06-26 Thread DImuthu Upeksha
Hi Lahiru, Thanks for the PR. I just merged it. For the error in django portal, you need to verify whether you can telnet to airavata.host:8443. If not, there is some issue in API Server. Please send API server logs as well Dimuthu On Sun, Jun 26, 2022 at 7:06 AM Lahiru Jayathilake <

Re: Repository and Front end Technology

2022-06-07 Thread DImuthu Upeksha
Hi Himanshu, We need to understand your designs and development plan before reviewing your code to make sure that we are on the same page. Please send a description of how and what you are going to implement for the benefit of others in this mail list who do not have a context Thanks Dimuthu On

[Airavata] Overriding and removing SLURM script parameters

2022-01-05 Thread DImuthu Upeksha
Hi All, With the introduction of GPU queues in various clusters, traditional SLURM scripts need several changes and those changes are not consistent with each cluster. Because of that, we need to override SLURM script parameters depending on the cluster configuration. This PR [1] provides the

[Airavata] Log4j migration and notes for future development

2021-12-16 Thread DImuthu Upeksha
Hi All, As you might all have heard, some versions of log4j, including 1.x and 2.x have been exposed to remote code execution attacks [1], and the recommendation to avoid this is to upgrade log4j to v2.16.0. Currently, Airavata is running on log4j 1.x, and this [2] pull request should migrate

Re: PyPI project ownership

2021-10-04 Thread DImuthu Upeksha
Hi Marcus, Sorry for missing the earlier invitation. I just accepted it. Thanks Dimuthu On Mon, Oct 4, 2021 at 6:00 PM Christie, Marcus Aaron wrote: > Looks like the invite expired. I've resent them. > > > On Sep 2, 2021, at 5:55 PM, Christie, Marcus Aaron > wrote: > > > > Hi Dimuthu,

Reconsidering SSH client library

2021-01-06 Thread DImuthu Upeksha
Hi Folks, We have been using JCraft for a long time until when we move to SSHJ [3] recently as the standard ssh client library of Airavata job manager. Main reasons for that migration are the lack of error handlers and implementation complexity of JCraft whereas SSHJ, provided cleanly defined

Re: Data Catalog into its own repo

2020-11-09 Thread DImuthu Upeksha
+1 On Mon, Nov 9, 2020 at 11:28 AM Christie, Marcus Aaron wrote: > +1 > > > On Nov 9, 2020, at 9:35 AM, Marru, Suresh wrote: > > > > Hi All, > > > > Airavata Experiment catalog evolved over time and though the replica > catalog and data product models are stand alone, they are buried to use >

Re: [em-center-gateway] MFT requirements for EMC portal

2020-10-21 Thread DImuthu Upeksha
Hi Dinuka, Sorry for the late reply. Please find my reply inline. Hope that helps. However it would be great if you can come up with a simple component / architecture diagram that visualizes how these components might connect to each other. It doesn't have to be perfect but atleast we might be

Re: HTCondor Extension Help

2020-10-20 Thread DImuthu Upeksha
ion to set the container’s hosts file, allowing me to > connect to the web interface presented by the container. I should note that > we have one developer doing this on native Ubuntu Linux and he faced the > same issue. > > > > Thank you! > > > > - Daniel Shchur >

Re: HTCondor Extension Help

2020-10-19 Thread DImuthu Upeksha
Any help with this would be greatly appreciated. Thanks! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha > *Sent: *Wednesday, October 14, 2020 12:33 AM > *To: *Airavata Dev > *Cc: *daniellthibod...@gmail.com; Evan Palmer > > *Subject: *Re:

Re: HTCondor Extension Help

2020-10-13 Thread DImuthu Upeksha
Daniel Shchur wrote: > Hi Dimuthu, > > > > Windows 10 2004 19041.508 > > Docker v19.03.13 running on WSL2 - Ubuntu 20.04 > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha > *Sent: *Tuesday, October 13, 2020 5:34 PM > *To: *Airavata D

Re: HTCondor Extension Help

2020-10-13 Thread DImuthu Upeksha
t/app/storage/views/16ac8ea2e3f88487e87226a07d300af0): > failed to open stream: No such file or directory > > > > > > > optimize [--force] [--psr] > > > > Script php artisan optimize handling the post-install-cmd event returned > with error code 1 > > Any help would b

Re: HTCondor Extension Help

2020-10-09 Thread DImuthu Upeksha
sion. Thank you! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha > *Sent: *Thursday, October 8, 2020 4:58 PM > *To: *Airavata Dev > *Cc: *daniellthibod...@gmail.com; Evan Palmer > > *Subject: *Re: HTCondor Extension Help > > > > Hi Daniel, > >

Re: HTCondor Extension Help

2020-10-08 Thread DImuthu Upeksha
we can let it know how to submit an > HTCondor job. > > Any help on this would be greatly appreciated. Thanks! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha > *Sent: *Monday, October 5, 2020 9:50 AM > *To: *Airavata Dev > *Cc: *daniellthibod...@g

Re: HTCondor Extension Help

2020-10-04 Thread DImuthu Upeksha
gt; Django portal and are curious what you would recommend, and how to do it > using your recommendation. > > > > Thanks! > > > > - Daniel Shchur > > > > *From: *DImuthu Upeksha > *Sent: *Friday, October 2, 2020 5:22 PM > *To: *Airavata Dev > *Cc:

Re: HTCondor Extension Help

2020-10-02 Thread DImuthu Upeksha
Hi Daniel, This is great progress. I think you are on the right track. Did you manage to set up the ide integration for the development of Airavata. If you have already done that, you can test these features. However I recommend you to send pull requests to a separate branch (htcondor?) and then

Re: Recursive entity fetching in registry

2020-09-22 Thread DImuthu Upeksha
ub.com/hasura/graphql-engine > [5] - https://hasura.io/docs/1.0/graphql/core/guides/mysql-preview.html > [6] - https://github.com/graphql/graphql-spec/issues/91 > [7] - > https://hasura.io/docs/1.0/graphql/core/schema/table-relationships/database-modelling/one-to-many.html > > >

Recursive entity fetching in registry

2020-09-22 Thread DImuthu Upeksha
Hi Folks, I just noticed that the entity relations that have bidirectional definitions (OneToMany and ManyToOne) between 2 tables cause recursive object fetching and it is causing a significant performance drop in database querying. For example [1] and [2]. When we fetch an experiment output, I

Re: Establishing an SSH connection to StorageResource

2020-08-19 Thread DImuthu Upeksha
Hi Vivek, The error you are getting is self explanatory. You are trying to SSH into a storage resource with a ssh key that is not available in the storage's authorized_keys file. I can see that you are trying to create a new ssh credential. token = generateAndRegisterSSHKeys(authzToken, "");

Re: Creating experiments on local

2020-08-06 Thread DImuthu Upeksha
Hi Vivek, Yes instructions should be updated in the wiki page. However, you can follow the instructions provided in [1] to deploy Airavata in the IDE. It should be upto date. [1] https://github.com/apache/airavata/tree/master/modules/ide-integration Thanks Dimuthu On Thu, Aug 6, 2020 at 3:19

Re: Updating the Airavata README.md

2020-08-05 Thread DImuthu Upeksha
Hi Marcus, This is very nice and clean. I suggested a minor change and what do you think if we point to ide integration module in the readme as a quick start point? Thanks Dimuthu On Wed, Aug 5, 2020 at 11:01 AM Christie, Marcus Aaron wrote: > Hi All, > > I've attempted to bring the Airavata

Re: [External] Re: Airavata MFT- Gdrive transport implementation

2020-05-01 Thread DImuthu Upeksha
April 30, 2020 12:43 PM > *To: *dev@airavata.apache.org > *Subject: *RE: [External] Re: Airavata MFT- Gdrive transport > implementation > > > > Hi Dimuthu, > > > > We have now addressed all the issues raised. > > Could you please verify and provide your inputs.

Re: [External] Re: Airavata MFT- Gdrive transport implementation

2020-04-28 Thread DImuthu Upeksha
know what > should be addressed further. > > Thanks and regards, > Aniruddha Patil > > On Apr 28, 2020, at 10:05 PM, DImuthu Upeksha > wrote: > > Thanks. I did the first pass of you PR. Please have a look. Once you > address the issues, I will do more detailed review

Re: Passwords and Passphrases

2020-04-28 Thread DImuthu Upeksha
ata-mft/pull/12 > > With regards, > Gopi Kiran > > > On Tue, Apr 28, 2020 at 12:27 AM DImuthu Upeksha < > dimuthu.upeks...@gmail.com> wrote: > >> Awesome. Looking forward for your PR >> >> On Tue, Apr 28, 2020 at 12:07 AM GOPI KIRAN >&

Re: [External] Re: Airavata MFT- Gdrive transport implementation

2020-04-28 Thread DImuthu Upeksha
ith the same name (it > creates a new version ). So how should this be handled? Should it create a > new file or update the existing file? > > > Let us know your inputs. > > > Regards, > > Nikita > -- > *From:* Bafna, Nikita > *Sent:* Tuesday, April 28, 2020

Re: Passwords and Passphrases

2020-04-27 Thread DImuthu Upeksha
t; I have finished the FTP implementation and will raise the PR by tomorrow. > If I need to implement the encryption part for the local DB, I will > implement it in the next PR. > > With regards, > Gopi Kiran > > > On Mon, Apr 27, 2020 at 11:47 PM DImuthu Upeksha < >

Re: [External] Apache Airavata MFT - OneDrive transport implementation

2020-04-27 Thread DImuthu Upeksha
for MS Graph. (this > is to avoid a user-dependent login) > Also, the repository you linked to is a third-party client. Is that > permissible? > > Thanks and regards, > Aniruddha Patil > > On Apr 27, 2020, at 10:39 PM, DImuthu Upeksha > wrote: > > Hi, Vivek, > > I

Re: Passwords and Passphrases

2020-04-27 Thread DImuthu Upeksha
Hi Gopi, This is implementation specific for secret backend type. If you are using the File Backend, you don't need to encrypt as it is the developer backend to test the framework so it needs to be simple as possible. But for production level backends, yes we should have them encrypted. Thanks

Re: [External] Apache Airavata MFT - OneDrive transport implementation

2020-04-27 Thread DImuthu Upeksha
Hi, Vivek, I'm little confused about the relation between graph auth and one drive. Can't we use a client id/ secret like this [1]? [1] https://github.com/isac322/OneDrive-SDK-java#1-construct-client-object Thanks Dimuthu On Mon, Apr 27, 2020 at 5:48 PM Patil, Aniruddha Vivek wrote: > > On

Re: [External] Re: Airavata MFT- Gdrive transport implementation

2020-04-27 Thread DImuthu Upeksha
gt; > We do not need to drive these with IU decisions, but good point these will > lead to potential use cases. On that note, looks like Google Drive will be > used for non-sensitive data and one drive for PHI data. So an eventual plan > for MFT to support both will be nice. I

Re: Airavata MFT- Gdrive transport implementation

2020-04-24 Thread DImuthu Upeksha
Because of the IUs decision of migration of Box data to Google Drive, I expect the use cases will come into MFT to handle drive transports. So from the usability perspective, I'd prefer gdrive over one drive. But it's up to you to decide. Once you have finalized on one, please update it in [1]

Re: [External] Re: Apache Airavata MFT : File Autotransfer

2020-04-24 Thread DImuthu Upeksha
r time > period and hence we wanted to be sure before we commit. > > Regards, > Aniruddha > > On Apr 23, 2020, at 5:20 PM, DImuthu Upeksha > wrote: > > I would say this project is an interesting one to work on if you have some > quality time to put on. And I can provide

Re: [External] Re: MFT- Dropbox Transport Implementation

2020-04-24 Thread DImuthu Upeksha
] > > > > Kind Regards > > Akshay Rajvanshi > > > > *From: *DImuthu Upeksha > *Reply-To: *"dev@airavata.apache.org" > *Date: *Thursday, April 23, 2020 at 14:47 > *To: *Airavata Dev > *Subject: *[External] Re: MFT- Dropbox Transport Implementation

Re: MFT enhancements

2020-04-23 Thread DImuthu Upeksha
1pSbtINUqlBFcy5UrOW0/edit?usp=sharing >>> >>> Dimuthu >>> >>> On Thu, Apr 23, 2020 at 2:48 PM GOPI KIRAN >>> wrote: >>> >>>> Absolutely! >>>> >>>> With regards, >>>> Gopi Kiran >>>> >>

Re: [External] Re: Apache Airavata MFT : File Autotransfer

2020-04-23 Thread DImuthu Upeksha
> On Thu, Apr 23, 2020 at 2:24 PM Patil, Aniruddha Vivek > wrote: > >> Hello Dimuthu, >> >> Here is a link to our proposal ( >> https://github.com/airavata-courses/NSA/wiki/Assignment-3-Proposal) >> It has more detailed information about this feature. >

Re: MFT enhancements

2020-04-23 Thread DImuthu Upeksha
If you are interested, mark it in https://docs.google.com/spreadsheets/d/1M7-Reda-pCi1l-TSSstI6Yi1pSbtINUqlBFcy5UrOW0/edit?usp=sharing Dimuthu On Thu, Apr 23, 2020 at 2:48 PM GOPI KIRAN wrote: > Absolutely! > > With regards, > Gopi Kiran > > > On Thu, Apr 23, 2020 at 2:

Re: MFT- Dropbox Transport Implementation

2020-04-23 Thread DImuthu Upeksha
If you set the md5 to null, it will not be evaluated. So you should be fine Dimuthu On Thu, Apr 23, 2020 at 2:41 PM Ravichandran, Sharanya wrote: > Hi, > > > We were taking a look at dropbox's documentation and dropbox's API doesnt > provide the md5checksum. > > > Since we have md5 integrity

Re: MFT enhancements

2020-04-23 Thread DImuthu Upeksha
Hi Gopi, Thanks for the interest. Globus and TUS are little complicated for a new developer for MFT to try out as there are some complicated message paths in both transports. I believe you can start with FTP and go through the pull request process to understand basics of implementing a transport

Re: [External] Re: Apache Airavata MFT : File Autotransfer

2020-04-23 Thread DImuthu Upeksha
gt; It has more detailed information about this feature. > > Regards, > Aniruddha > > On Apr 22, 2020, at 10:42 PM, DImuthu Upeksha > wrote: > > This message was sent from a non-IU address. Please exercise caution when > clicking links or opening attachments from external source

Re: Apache Airavata MFT : File Autotransfer

2020-04-22 Thread DImuthu Upeksha
Hi Aniruddha, Can you note down more fine grained goals of this feature. May be add some use cases as well. Then I will be able to help in figuring out where to implement it. Thanks Dimuthu On Wed, Apr 22, 2020 at 9:37 PM Patil, Aniruddha Vivek wrote: > Hello, > As per discussions with Prof.

Re: [External] Re: Apache Airavata MFT - AWS/GCS support

2020-04-22 Thread DImuthu Upeksha
ached the screenshot for your reference. > > > > Could you please help us with this? > > > > Thank you > > Aravind Ramalingam > > > > > > > > On Mon, Apr 20, 2020 at 10:59 PM Aravind Ramalingam > wrote: > > Hi Dimuthu, > > > >

Re: Queries on MFT containerization

2020-04-19 Thread DImuthu Upeksha
Hi Shivam, I will reply inline On Sun, Apr 19, 2020 at 12:06 PM Rastogi, Shivam wrote: > Hi Team, > > > For our final project, we are trying to containerize and deploy MFT > services in a service-mesh. > > > Could you please advise on the following. > > > 1. Why did we use Consul in MFT,

MFT Transport Implementation Status

2020-04-19 Thread DImuthu Upeksha
Hi Folks, I'm really glad to see some of you have already started working on implementing transports for MFT. So I thought of sharing a spreadsheet [1] to track who is doing what to make sure that two persons are not working on the same transport. Please go ahead and update the document if you

Re: [airavata-mft] Box transport agent for airavata mft

2020-04-19 Thread DImuthu Upeksha
Hi Dinuka, I reviewed your pending PR and it looks really good. I'm glad that you have followed standard coding style to make this more consistent with the rest of the code. I will reply inline for your comments [9] https://github.com/apache/airavata-mft/pull/5 Thanks Dimuthu On Sun, Apr 19,

Re: [airavata-mft] Separating mft-agent module to a golang implementation

2020-04-19 Thread DImuthu Upeksha
Hi Dinuka, I will reply inline On Fri, Apr 17, 2020 at 12:37 PM Dinuka Desilva wrote: > Hi, > > I'm currently looking in to convert the existing java implementation of > agent module to golang considering better concurrency, performance and make > it more platform enabled. Before just starting

Re: [External] Re: Apache Airavata MFT - AWS/GCS support

2020-04-19 Thread DImuthu Upeksha
file has to be present and > converted into a single stream and uploaded at once? > > We had understood that MFT expects it to be chunk by chunk upload without > having to have the entire file present. > > > > Thank you > > Aravind Ramalingam > > > > On Apr

Re: Apache Airavata MFT - AWS/GCS support

2020-04-16 Thread DImuthu Upeksha
>> will take up and explore GCS. >> >> Thank you for the update. >> >> Thank you >> Aravind Ramalingam >> >> On Apr 16, 2020, at 00:30, DImuthu Upeksha >> wrote: >> >>  >> Aravind, >> >> I'm not sure whether you have made

Re: Apache Airavata MFT - AWS/GCS support

2020-04-15 Thread DImuthu Upeksha
ok into > Azure. > > Thank you > Aravind Ramalingam > > On Mon, Apr 6, 2020 at 4:44 PM DImuthu Upeksha > wrote: > >> Aravind, >> >> Here [2] is the complete commit for S3 transport implementation but don't >> get confused by the amount of changes as

Re: [airavata-mft] Null pointer exception on LOCAL to LOCAL transport

2020-04-15 Thread DImuthu Upeksha
Hi Dinuka, It should be fixed by now. Local transport was not enhanced for the latest architecture. This [1] commit should fix it [1] https://github.com/apache/airavata-mft/commit/380a674d904ceab21a32c353b74c3742472801ec Thanks Dimuthu On Wed, Apr 15, 2020 at 4:32 PM Dinuka Desilva wrote: >

Re: [airavata-django-portal] Cannot keep the localhost db running for more than half a hour

2020-04-09 Thread DImuthu Upeksha
certificate in case if we need it in future. [2] https://github.com/apache/airavata/tree/develop/modules/ide-integration#note-optional-creating-certificates-if-expired Thanks Dimuthu On Thu, Apr 9, 2020 at 9:26 PM DImuthu Upeksha wrote: > Dinuka > > I saw couple of issues but

Re: [airavata-django-portal] Cannot keep the localhost db running for more than half a hour

2020-04-09 Thread DImuthu Upeksha
w?usp=drive_web> > Regards, > Dinuka > > On Fri, Apr 10, 2020 at 5:00 AM DImuthu Upeksha < > dimuthu.upeks...@gmail.com> wrote: > >> Hi Dinuka, >> >> Can you please resend the screenshot you have sent before? I guess apache >> mail server drops a

Re: [airavata-django-portal] Cannot keep the localhost db running for more than half a hour

2020-04-09 Thread DImuthu Upeksha
I'm using the "develop" branch on both repos. (airavata and > airavata-django-portal) > > Regards, > Dinuka > > On Thu, Apr 9, 2020 at 2:15 PM DImuthu Upeksha > wrote: > >> Dinuka, >> >> What is the branch you are using? >> >> Dimuthu

Re: [airavata-django-portal] Cannot keep the localhost db running for more than half a hour

2020-04-09 Thread DImuthu Upeksha
Dinuka, What is the branch you are using? Dimuthu On Wed, Apr 8, 2020 at 2:45 PM Dinuka Desilva wrote: > Hi, > > Let me upload the logs of docker-compose, airavata-server and > airavata-django-portal below. I've been actually having this issue > continuously and not once in a while. Most of

Re: Apache Airavata MFT - AWS/GCS support

2020-04-06 Thread DImuthu Upeksha
ata-mft/blob/master/services/secret-service/stub/src/main/proto/SecretService.proto#L45 Thanks Dimuthu On Sun, Apr 5, 2020 at 12:10 AM DImuthu Upeksha wrote: > There is a working on S3 transport in my local copy. Will commit it once I > test it out properly. You can follow the same pattern for any

Re: Question regarding Resource and Secret service Backend in Production

2020-04-04 Thread DImuthu Upeksha
Adding Isuru explicitly to the discussion Good question. This is something that we need to properly evaluate the margin of integration between Custos and general Airavata data models. I agree that resources are not directly integrated with Custos but at some point, Resource Service has to talk to

Re: Apache Airavata MFT - AWS/GCS support

2020-04-04 Thread DImuthu Upeksha
There is a working on S3 transport in my local copy. Will commit it once I test it out properly. You can follow the same pattern for any cloud provider which has clients with streaming IO. Streaming among different transfer protocols inside an Agent has been discussed in the last part of this [1]

Setting up MFT in local envornments

2020-04-02 Thread DImuthu Upeksha
Hi folks This is specially for GSoC applicants who started working in Airavata Managed File Transfer project. First of all I should apologize for the confusion happened in setting up MFT in your local environments. This a very new project and we are working on specific goals internally so some

Re: [External] Airavata MFT project errors

2020-04-02 Thread DImuthu Upeksha
ion? > > Also, in Secret service the application.properties file has localhost for > credentials server. > [image: image.png] > Are we supposed to run any other project to support the service? > > Thank you > Aravind Ramalingam > > On Thu, Apr 2, 2020 at 4:04 PM

Re: [External] Airavata MFT project errors

2020-04-02 Thread DImuthu Upeksha
tos > project for it's data. > > Thank you > Aravind Ramalingam > > On Apr 2, 2020, at 15:24, DImuthu Upeksha > wrote: > >  > Hi Aravind, > > Yes your understanding is correct. We assume that Resource service and > Secret Service has all the data you need to initiat

Re: [External] Airavata MFT project errors

2020-04-02 Thread DImuthu Upeksha
ed. > > Thank you > Aravind Ramalingam > > On Apr 2, 2020, at 12:26, DImuthu Upeksha > wrote: > >  > Hi Aravind, > > Currently all the calls to the API are performed through gRPC. For Java, > clients are already generated from the maven build, but if you are usi

Re: MFT and data access for running jobs

2020-03-26 Thread DImuthu Upeksha
rovide access from > gateway storage or directly to users desktop periodically/or on prompt > during the run. > > > > Thanks, > > Sudhakar. > > > > *From: *DImuthu Upeksha > *Reply-To: *"dev@airavata.apache.org" > *Date: *Thursday, March 26, 2

Re: MFT and data access for running jobs

2020-03-26 Thread DImuthu Upeksha
the > launch of the experiment is registered as a remote disk accessible > > The contents are made available readonly for users to read/download > > Remove this as accessible when the experiment ends > > Continue with the rest of the Helix tasks > > … > > >

Re: MFT and data access for running jobs

2020-03-26 Thread DImuthu Upeksha
Sudhakar, I’m not sure whether I grabbed your point about this remote working directory correctly. Are you taking about the working directory of the cluster? Can you please explain the workflow with more details? Thanks Dimuthu On Thu, Mar 26, 2020 at 10:21 AM Pamidighantam, Sudhakar wrote: >

Re: Upgrade Java Version

2019-12-06 Thread DImuthu Upeksha
e great to get the Travis build working again. > > On Oct 21, 2019, at 4:07 PM, DImuthu Upeksha > wrote: > > Hi Folks, > > What should we do about this? Should we contact Apache legal support to > get things clarified about the licensing? We might need to stick to a > decision before next release. > > Thanks > Dimuthu > > > >

Re: Task is running periodically

2019-11-25 Thread DImuthu Upeksha
no activity in the cluster. > > As for the ConcurrentModification exception, I believe there was a patch > for that going from 0.8.2 to the latest on the upstream. > > Hunter > > On Mon, Nov 25, 2019 at 10:13 AM DImuthu Upeksha < > dimuthu.upeks...@gmail.com> wrote: &

Task is running periodically

2019-11-25 Thread DImuthu Upeksha
Hi Folks, We have noticed a task in Helix cluster was running periodically even though it was completed at each run. When we look at the logs of the controller, I can see some Concurrent Modification exceptions [1]. However this is a very rare occurrence. We have been using current Helix version

Re: Upgrade Java Version

2019-10-21 Thread DImuthu Upeksha
Hi Folks, What should we do about this? Should we contact Apache legal support to get things clarified about the licensing? We might need to stick to a decision before next release. Thanks Dimuthu

Re: 0.18 Release Planning

2019-10-21 Thread DImuthu Upeksha
Hi Suresh, I don't have any objection to create the 0.18 branch now. As the production deployments are stable, I don't hope that it will require any major updates in near future. This will even give us enough buffer to stabilize master after merging the staging. Thanks Dimuthu On Mon, Oct 21,

Re: Upgrade Java Version

2019-09-15 Thread DImuthu Upeksha
13, 2019 at 11:58 AM Christie, Marcus Aaron wrote: > +1 for Java 11 > > On Sep 12, 2019, at 10:28 AM, Suresh Marru wrote: > > + 1 for Java 11. > > Any objections to move forward to make Java 11 minimum requirement? > > Suresh > > On Sep 10, 2019, at 6:06 PM,

Re: Upgrade Java Version

2019-09-10 Thread DImuthu Upeksha
Hi Suresh, I would vote for Java 11 as it's relatively mature and a LTS version [1] [1] https://www.oracle.com/technetwork/java/java-se-support-roadmap.html Thanks Dimuthu On Thu, Sep 5, 2019 at 8:34 AM Suresh Marru wrote: > Hi All, > > I noticed a travis build fail which complains minimum

Re: Class not found error occurred when executing "registerSample.sh" in airavata-server-0.17/samples

2019-06-19 Thread DImuthu Upeksha
Hi Isuru, Are you following any tutorial to get started with Airavata? Can you please share more details. Some of them are outdated and we are working on making them better. If you want to get started with Airavata, I would recommend you to go with IDE Integration [1] module where you can setup

Re: Clarification on task level timeouts

2019-06-05 Thread DImuthu Upeksha
ed, it will have a timeout of 30 > minutes, and if it hasn't completed after 30 minutes, only that task will > be timed out, not its parent job. > > Hope that helps, > Hunter > > On Wed, Jun 5, 2019 at 6:44 AM DImuthu Upeksha > wrote: > >> Hi Folks, >> >> I have seen s

Clarification on task level timeouts

2019-06-05 Thread DImuthu Upeksha
Hi Folks, I have seen some tasks are getting cancelled automatically when they are running more than 1 hour. I feel like the controller is doing that based on some sort of a timeout. Can we remove that timeout? I saw 2 api methods that are related to this problem but I don't have a clear Idea on

Re: Zookeeper connection errors in Helix Controller

2019-05-31 Thread DImuthu Upeksha
) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041) Thanks Dimuthu On Fri, May 31, 2019 at 1:14 PM DImuthu Upeksha wrote: > Hi Lei, > > We use 0.8.2. We initially had 0.8.4 but it contains an issue with task > retry

Re: Zookeeper connection errors in Helix Controller

2019-05-31 Thread DImuthu Upeksha
, 2019 at 1:04 PM Lei Xia wrote: > Which Helix version do you use? This may caused by this Zookeeper bug ( > https://issues.apache.org/jira/browse/ZOOKEEPER-706). We have upgraded > ZkClient in later Helix versions. > > > Lei > > On Fri, May 31, 2019 at 7:52 AM DImuthu U

Re: Zookeeper connection errors in Helix Controller

2019-05-31 Thread DImuthu Upeksha
, 2019 at 12:25 PM DImuthu Upeksha wrote: > Hi Kishore, > > Please find below log and I think the issue is "Packet len4194362 is out > of range!". Currently we have around 1 unprocessed workflows. Will that > be the reason for that? > > 2019-05-31 12:17:15,

Re: Zookeeper connection errors in Helix Controller

2019-05-31 Thread DImuthu Upeksha
read] INFO o.a.h.m.zk.zookeeper.ZkClient - zookeeper state changed (Disconnected) Thanks Dimuthu On Fri, May 31, 2019 at 11:55 AM kishore g wrote: > can you grep for zookeeper state in controller log. > > On Fri, May 31, 2019 at 7:52 AM DImuthu Upeksha < > dimuthu.upeks...@gma

Zookeeper connection errors in Helix Controller

2019-05-31 Thread DImuthu Upeksha
Hi Folks, I'm getting following error in controller log and seems like controller is not moving froward after that point 2019-05-31 10:47:37,084 [main] INFO o.a.a.h.i.c.HelixController - Starting helix controller 2019-05-31 10:47:37,089 [main] INFO o.a.a.c.u.ApplicationSettings - Settings

Cleaning up unused code

2019-04-29 Thread DImuthu Upeksha
Hi Folks, I'm working on $subject and reducing the size of distributions with minimal dependencies. I will keep adding the classes that I'm going to remove to this thread and please let me know whether those are required.

Re: Submitted the project draft - Metascheduler for Airavata

2019-04-08 Thread DImuthu Upeksha
Hi Bibrak, Please go through [1] this mail thread and capture current use cases for new scheduler. You can summarize them in the proposal. [1] https://www.mail-archive.com/dev@airavata.apache.org/msg12180.html Thanks Dimuthu On Mon, Apr 8, 2019 at 2:44 PM Bibrak Qamar wrote: > Hi all, > > I

Re: ZkHelixManager disconnection hangs

2019-04-02 Thread DImuthu Upeksha
PM Wang Jiajun > wrote: > >> Hi Dimuthu, >> >> Did you stop the controller when the connection is flapping or when it is >> normal? >> Could you please list all the steps that you have done in order? >> >> Best Regards, >> Jiajun >> >> >

Re: Sporadic delays in task execution

2019-03-24 Thread DImuthu Upeksha
g tasks, so you > > might want to consider using the newest version as well. > > > > Hunter > > > > On Fri, Mar 22, 2019 at 8:59 AM DImuthu Upeksha < > > dimuthu.upeks...@gmail.com> wrote: > > > >> Hi Lee, > >> > >> Thanks

Re: Sporadic delays in task execution

2019-03-22 Thread DImuthu Upeksha
nce/schedule things > > On Thu, Mar 21, 2019 at 11:30 AM DImuthu Upeksha < > dimuthu.upeks...@gmail.com> wrote: > >> Hi Junkai, >> >> What do you mean by touching ideal state to trigger an event? I didn't >> quite get what you said. Is that like creating some

Re: Sporadic delays in task execution

2019-03-21 Thread DImuthu Upeksha
Xue Junkai wrote: > Can you try one thing? Touch the ideal state to trigger an event. If > workflows are not scheduled, it should scheduling has problem. > > Best, > > Junkai > > On Wed, Mar 20, 2019 at 10:31 PM DImuthu Upeksha < > dimuthu.upeks...@gmail.com>

Re: [VOTE] Apache Airavata release 0.17 - RC1

2019-03-21 Thread DImuthu Upeksha
+1 On Thu, Mar 21, 2019 at 1:07 AM Suresh Marru wrote: > Apache Airavata PMC is pleased to call for a vote on the following Apache > Airavata 0.17 release candidate artifacts: > > Detailed change log/release > notes:https://github.com/apache/airavata/blob/airavata-0.17/RELEASE_NOTES > > All

Re: [DISCUSS] Apache Airavata release 0.17 - RC1

2019-03-21 Thread DImuthu Upeksha
ately after. We can put a note on > downloads page. > > Cheers, > Suresh > > On Mar 21, 2019, at 1:54 AM, DImuthu Upeksha > wrote: > > Hi Suresh, > > Build succeeded in JDK 1.8 with Maven 3.5.2 on Mac OS. INSTALL file in the > binary distribution looks little bit

Re: [DISCUSS] Apache Airavata release 0.17 - RC1

2019-03-20 Thread DImuthu Upeksha
Hi Suresh, Build succeeded in JDK 1.8 with Maven 3.5.2 on Mac OS. INSTALL file in the binary distribution looks little bit out dated. Thanks Dimuthu On Thu, Mar 21, 2019 at 1:07 AM Suresh Marru wrote: > Discussion thread for vote on Apache Airavata 0.17 release candidate. > > If you have any

Re: Sporadic delays in task execution

2019-03-20 Thread DImuthu Upeksha
Hi Junkai, We are using 0.8.1 Dimuthu On Thu, Mar 21, 2019 at 12:14 AM Xue Junkai wrote: > Hi Dimuthu, > > What's the version of Helix you are using? > > Best, > > Junkai > > On Wed, Mar 20, 2019 at 8:54 PM DImuthu Upeksha < > dimuthu.upeks...@gmail

  1   2   3   >