RE: Progress so far on the website

[Miller, Mark]

Hi Davis,

I did not see much difference between the two screenshots.
So for an admin, I would expect to see prominently tools to manage accounts, 
and to see what users are doing.
But a lot for the admin depends on what tasks the admin is charged with
I do think the tools to accept and reject discussion items are relevant for 
admin.

For students, at least, I would tend to separate discussions into their own 
area/tab, and perhaps for admins as well.
I am easily distracted personally, and don’t like sites that bombard me with 
many ways to go when I come specifically to do work.
The four applications are given a lot of real estate, and I think this is 
unnecessary for the admin, she knows what they are.

For the student, this may work ok, if the number of apps is small. But if it 
grows to twenty you will/may have issues.
For a student I would expect prominent help areas, and a tab for how to use 
this maybe. I don’t know the workflow, but I think a “data” area would also be 
prominent,
For uploaded data that users can add to their jobs.

Again I don’t know the workflow, but I am not sure what will happen if I click 
application. Will I get a description, or will it spawn a window to configure a 
job. If the latter, where will the data come from? Will it be stored on the 
site? If the former, how to I get from the app to configuring jobs?
I might prefer an interface that has a “create a job” button, and then figure 
it out from there.

Those are my quick thoughts.

Mark

From: Davis Joseph [mailto:datjo...@iu.edu]
Sent: Saturday, July 28, 2018 9:05 PM
To: dev@airavata.apache.org
Subject: Progress so far on the website

This will only take a second or so,

I have made the admin and student dashboard but I want to hear some feedback 
about whether or not you personally would use this website if you were a 
student studying weather and climate. Or if you have any idea on how to improve 
the site.

The discussions section will work like StackOverflow.

I've taken screen shots of the website layout and attached them.


Davis Joseph

RE: Welcome Marcus Christie as Airavata PMC member

[Miller, Mark]

Congratulations, and welcome!

-Original Message-
From: Suresh Marru [mailto:sma...@apache.org] 
Sent: Friday, November 17, 2017 11:11 AM
To: Airavata Dev 
Cc: Airavata Users 
Subject: Welcome Marcus Christie as Airavata PMC member

Hi All,

The Project Management Committee (PMC) for Apache Airavata has asked Marcus 
Christie to become a PMC member based on his contributions to the project. We 
are pleased to announce that he has accepted.

As you know, Marcus has been stewarding Airavata already as a committer and 
being a PMC member will enable him to assist with the management and to guide 
the direction of the project as well.

Please join me in welcoming Marcus to Airavata PMC

Cheers,
Suresh
(On Behalf of Apache Airavata PMC)

RE: [#Spring17-Airavata-Courses] Reliable file uploads for Science Gateway Portals

[Miller, Mark]

Hi Ameya,

We have thought about this at CIPRES as well.
Terri Schwartz in our group has implemented a Java version of resumable file 
transfer in tus.io, and I have mentioned this as a possible SciGaP/Airavata 
service. I would talk recommend you talk with Terri and see if there is synergy 
between your thinking, and what she has done.

Mark

From: Ameya Advankar [mailto:aadva...@umail.iu.edu]
Sent: Sunday, May 07, 2017 9:23 PM
To: dev@airavata.apache.org
Subject: [#Spring17-Airavata-Courses] Reliable file uploads for Science Gateway 
Portals

Hi Airavata Developers,

I have been exploring and evaluating tus.io  as a solution to 
the following problems encountered in Science Gateway Portals related to the 
file upload functionality:

1. Unreliable HTTP Connection

Since the File uploads in Science Gateway Portals are HTTP uploads, these are 
heavily reliant on a continuous internet connection being available on the 
client machine. There could be network disruptions or connectivity issues and a 
traditional file upload will fail in this case. As a result, the users may have 
to retry the uploads manually and wait for a successful upload to take place. 
If the files are large i.e. a few hundred Megabytes or some Gigabytes, this 
will cause a waste of bandwidth and time.

2. Space constraints on the Server

The file which is being uploaded usually would be staged somewhere on the 
Server for a certain period of time till it is picked up for further 
processing. The file may be there for a considerable amount of time depending 
on the process queuing time. In case multiple large files are uploaded at the 
same time by users of a Portal, the host machine may run out of space and this 
could have adverse affects on the performance of the Portal.

Also, there could be cases in which multiple Science Gateway Portals are hosted 
on a the same web server. In such a setting, if a particular Portal fills up 
server space with multiple large files, it may affect the performance of other 
Portals residing on that web server as well. In short, the file-upload 
functionality should not affect the Portal performance.


We could use some JavaScript libraries such as Fine 
Uploader, Resumable.js  
or flow.js which provide a simple client 
side library for resumable file uploads in case of disruptions. Fine Uploader 
seems to be the best among these libraries based on the community usage and 
contribution on Github. For each of these client side libraries, we have to 
incorporate the corresponding server side code to handle resumable uploads.

However since their JavaScript implementations are unique, with each library 
using their own set of parameters and request headers to achieve Resumable 
functionality, the Server implementation which we adopt will be tightly coupled 
with the library we choose. This will introduce a dependency on the library.
To remove the library based dependency, we can use a client-server 
implementation of tus.io protocol. Using a protocol will reduce 
the library level dependency to a protocol level dependency.

Also since tus.io client could be implemented in any language, 
we could have multiple types of Gateway Portals such as Web, Desktop and native 
which connect to the same tus.io based server.

The Second problem of space constraint can be solved by separating the 
file-upload process as a micro service located on a separate host. Each Portal 
could have its own separate micro-service instance and this way the file-upload 
functionality will not hamper the Portal performance. Further, the 
micro-service will have to be secured and tus.io allows us to do 
this via the tus.io 
hooks feature by 
implementing the Auth code in the pre-create hook.

Thus, tus.io seems to be a good, flexible and maintainable 
solution for implementing file-upload functionality in Science Gateway Portals 
from a long term perspective.

Thanks & Regards,
Ameya Advankar
Masters in Computer Science,
Indiana University Bloomington

RE: Database Version Control for Airavata

[Miller, Mark]

Thanks Anuj, For taking the time to send that, it was super helpful, and 
answered my questions.
Mark

From: Anuj Bhandar [mailto:bhandar.a...@gmail.com]
Sent: Tuesday, December 27, 2016 10:23 AM
To: dev@airavata.apache.org
Subject: Re: Database Version Control for Airavata


HI Mark,

Liquibase is a Database Version control tool, it gives you the ability to 
rollback any changes (commit) on DataBase Schema just like GIT, but it will 
only be useful for future builds, i.e. if you implement Liquibase in current 
build, make changes on top of that, then those changes can be rolled back, in 
short the version to be rolled back should be present in the repository. Works 
just like GIT for source code.

But if you are referring to Airavata's scenario, i.e. we have two version of DB 
schema's, one for Development and the other for Production, then a tool like 
DBDiff (https://github.com/DBDiff/DBDiff) compares the two different versions 
and generates Delta SQL scripts, which can be run on the DB schema's, DBDiff 
has the ability to generate both UP and Down Migration scripts.

I hope that I have answered the question, please revert back if you need 
further details.
Thanks and best regards,
Anuj Bhandar
MS Computer Science
Indiana University Bloomington
+1 812 361 9414
On 12/27/2016 9:19 AM, Christie, Marcus Aaron wrote:
Mark,

Anuj might know more about this but I thought I would chime in. I believe this 
is something that database migration tools can do. I haven’t tried that though. 
Here’s Liquibase’s support for rollback scripts: 
http://www.liquibase.org/documentation/rollback.html

Marcus

On Dec 25, 2016, at 8:49 AM, Miller, Mark 
mailto:mmil...@sdsc.edu>> wrote:

Hi Anuj,

Very interesting. I had a question, rather than a vote. We have found (at 
CIPRES) that we really wish we had the ability to make DB schema changes and 
test them and roll them back if there is an issue with what we tried. I wonder 
if this is something the new or existing DBs can do in AIravata?

Mark

From: Pierce, Marlon [mailto:marpi...@iu.edu]
Sent: Friday, December 23, 2016 10:11 AM
To: dev@airavata.apache.org<mailto:dev@airavata.apache.org>
Subject: Re: Database Version Control for Airavata

+1 for looking into Liquibase. I’d be interested to hear other opinions.

Marlon

From: Anuj Bhandar mailto:bhandar.a...@gmail.com>>
Reply-To: "dev@airavata.apache.org<mailto:dev@airavata.apache.org>" 
mailto:dev@airavata.apache.org>>
Date: Friday, December 23, 2016 at 12:55 PM
To: "dev@airavata.apache.org<mailto:dev@airavata.apache.org>" 
mailto:dev@airavata.apache.org>>
Subject: Database Version Control for Airavata

Hello Dev,
Merry Christmas ! Hope you guys are doing good.
As a part of the Airavata 0.17 release, we are faced with a industry standard 
problem of migrating our old master database to the new Development database, 
the effort needed to manually write migration SQL scripts is huge.
Hence for the current release, I'm relying on DBDiff 
(http://dbdiff.github.io/DBDiff/) for generating Delta SQL scripts for the 
entire database and manually write scripts for complex DB schema changes.
To address the issue in future releases, I propose to implement a database 
version control system like Liquibase (http://www.liquibase.org/) for Airavata, 
this will help automate the creation of delta scripts and also adding 
additional functionality like backward compatibility between releases.
Please feel to suggest changes and also comment about the products I'm using to 
accomplish these tasks.
Thanks and best regards,
Anuj Bhandar
MS Computer Science
Indiana University Bloomington
+1 812 361 9414

RE: Database Version Control for Airavata

[Miller, Mark]

Hi Anuj,

Very interesting. I had a question, rather than a vote. We have found (at 
CIPRES) that we really wish we had the ability to make DB schema changes and 
test them and roll them back if there is an issue with what we tried. I wonder 
if this is something the new or existing DBs can do in AIravata?

Mark

From: Pierce, Marlon [mailto:marpi...@iu.edu]
Sent: Friday, December 23, 2016 10:11 AM
To: dev@airavata.apache.org
Subject: Re: Database Version Control for Airavata

+1 for looking into Liquibase. I’d be interested to hear other opinions.

Marlon

From: Anuj Bhandar mailto:bhandar.a...@gmail.com>>
Reply-To: "dev@airavata.apache.org" 
mailto:dev@airavata.apache.org>>
Date: Friday, December 23, 2016 at 12:55 PM
To: "dev@airavata.apache.org" 
mailto:dev@airavata.apache.org>>
Subject: Database Version Control for Airavata


Hello Dev,

Merry Christmas ! Hope you guys are doing good.

As a part of the Airavata 0.17 release, we are faced with a industry standard 
problem of migrating our old master database to the new Development database, 
the effort needed to manually write migration SQL scripts is huge.
Hence for the current release, I'm relying on DBDiff 
(http://dbdiff.github.io/DBDiff/) for generating Delta SQL scripts for the 
entire database and manually write scripts for complex DB schema changes.

To address the issue in future releases, I propose to implement a database 
version control system like Liquibase (http://www.liquibase.org/) for Airavata, 
this will help automate the creation of delta scripts and also adding 
additional functionality like backward compatibility between releases.

Please feel to suggest changes and also comment about the products I'm using to 
accomplish these tasks.

Thanks and best regards,

Anuj Bhandar
MS Computer Science
Indiana University Bloomington
+1 812 361 9414

RE: mesos and moving jobs between clusters

[Miller, Mark]

Hi Folks,

Thanks for your kind answers.
We are very specifically interested in how Mesos might allow us to submit to 
multiple machines without changing the “rules” we impose that are machine 
dependent.
For example, Gordon has 16 cores per node and no gpu
Comet has 24 cores per node, and 4 gpu nodes.
So we adjust our running rules depending on the resource.

I can imagine one solution would be to find the least common denominator, and 
run all jobs
On virtual clusters with 16 cores maximum, and no gpus.
Or we might only submit jobs that can use gpus to resources that have them.

Anyway, since we will all be in SD soon, it seems like we should chat a bit 
about this in person?.
Maybe we can find a coffee break time, or set up a meeting during the meeting?

Mark




From: Pankaj Saha [mailto:psa...@binghamton.edu]
Sent: Tuesday, October 25, 2016 3:10 PM
To: dev 
Subject: Re: mesos and moving jobs between clusters


Hi Mark,


Mesos collects the resource information from all the nodes in the cluster 
(cores, memory, disk, and gpu) and presents a unified view, as if it is a 
single operating system. The Mesosphere, who a commercial entity for Mesos, has 
built an ecosystem around Mesos as the kernel called the "Data Center Operating 
System (DCOS)".  Frameworks interact  with Mesos to reserve resources and then 
use these resources to run jobs on the cluster. So, for example, if multiple 
frameworks such as Marathon, Apache Aurora, and a custom-MPI-framework are 
using Mesos, then there is a negotiation between Mesos and each framework on 
how many resources each framework gets. Once the framework, say Aurora, gets 
resources, it can decide how to use those resources. Some of the strengths of 
Mesos include fault tolerance at scale and the ability to co-schedule 
applications/frameworks on the cluster such that cluster utilization is high.


Mesos off-the-shelf only works when the Mater and agent nodes have a line of 
communication to each other. We have worked on modifying the Mesos installation 
so that it even works when agents are behind firewalls on campus clusters. We 
are also working on getting the same setup to work on Jetstream and Chameleon 
where allocations are a mix of public IPs and internally accessible nodes. This 
will allow us to use Mesos to meta-schedule across clusters. We are also 
developing our own framework, to be able to customize scheduling and resource 
negotiations for science gateways on Mesos clusters. Our plan is to work with 
Suresh and Marlon's team so that it works with Airavata.


I will be presenting at the Gateways workshop in November, and then I will also 
be at SC along with my adviser (Madhu Govindaraju), if you would like to 
discuss any of these projects.


We are working on packaging our work so that it can be shared with this 
community.



Thanks

Pankaj

On Tue, Oct 25, 2016 at 11:36 AM, Mangirish Wagle 
mailto:vaglomangir...@gmail.com>> wrote:
Hi Mark,

Thanks for your question. So if I understand you correctly, you need kind of 
load balancing between identical clusters through a single Mesos master?

With the current setup, from what I understand, we have a separate mesos 
masters for every cluster on separate clouds. However, its a good investigative 
topic if we can have single mesos master targeting multiple identical clusters. 
We have some work ongoing to use a virtual cluster setup with compute resources 
across clouds to install mesos, but not sure if that is what you are looking 
for.

Regards,
Mangirish


[https://mailtrack.io/trace/mail/8481e3ada5019bfcebef25b672239785318edb23.png?u=765734]

On Tue, Oct 25, 2016 at 11:05 AM, Miller, Mark 
mailto:mmil...@sdsc.edu>> wrote:
Hi all,

I posed a question to Suresh (see below), and he asked me to put this question 
on the dev list.
So here it is. I will be grateful for any comments about the issues you all are 
facing, and what has come up in trying this, as
It seems likely that this is a much simpler problem in concept than it is in 
practice, but its solution has many benefits.

Here is my question:
A group of us have been discussing how we might simplify submitting jobs to 
different compute resources in our current implementation of CIPRES, and how 
cloud computing might facilitate this. But none of us are cloud experts. As I 
understand it, the mesos cluster that I have been seeing in the Airavata email 
threads is intended to make it possible to deploy jobs to multiple virtual 
clusters. I am (we are) wondering if Mesos manages submissions to identical 
virtual clusters on multiple machines, and if that works efficiently.

In our implementation, we have to change the rules to run efficiently on 
different machines, according to gpu availability, and cores per node. I am 
wondering how Mesos/ virtual clusters affect those considerations.
Can mesos create basically identical virtual clusters independent of machine?

Thanks for any advice.

Mark

mesos and moving jobs between clusters

[Miller, Mark]

Hi all,

I posed a question to Suresh (see below), and he asked me to put this question 
on the dev list.
So here it is. I will be grateful for any comments about the issues you all are 
facing, and what has come up in trying this, as
It seems likely that this is a much simpler problem in concept than it is in 
practice, but its solution has many benefits.

Here is my question:
A group of us have been discussing how we might simplify submitting jobs to 
different compute resources in our current implementation of CIPRES, and how 
cloud computing might facilitate this. But none of us are cloud experts. As I 
understand it, the mesos cluster that I have been seeing in the Airavata email 
threads is intended to make it possible to deploy jobs to multiple virtual 
clusters. I am (we are) wondering if Mesos manages submissions to identical 
virtual clusters on multiple machines, and if that works efficiently.

In our implementation, we have to change the rules to run efficiently on 
different machines, according to gpu availability, and cores per node. I am 
wondering how Mesos/ virtual clusters affect those considerations.
Can mesos create basically identical virtual clusters independent of machine?

Thanks for any advice.

Mark

RE: Personal compute/storage preferences for campus portals

[Miller, Mark]

Hi Marcus, this sounds quite interesting. Do you mean that only users with this 
role will see the tabs for adding resource in their UI?
Would the title of resource_owner be descriptive?

Mark



From: Christie, Marcus Aaron [mailto:machr...@iu.edu]
Sent: Monday, October 10, 2016 11:06 AM
To: dev@airavata.apache.org
Subject: Personal compute/storage preferences for campus portals


Hello All,



I don't think I've written to this list yet, so let me introduce myself. My 
name is Marcus Christie and I work in the Science Gateways Group at IU with 
Suresh and other Airavata developers.  I'm looking forward to contributing to 
Airavata.



I'm currently working on creating a UI in PGA for a portal user to add their 
own compute and/or storage resource allocations 
(https://issues.apache.org/jira/browse/AIRAVATA-2117​). I recently met with 
Suresh and Eroma to discuss some UI concerns with how this will impact existing 
users of PGA that are today using the gateways allocation.  The problem is it 
could be confusing for users who don't have their own compute/storage 
allocations to see the new options in PGA for adding compute/storage 
allocations.  Also there are some additional UI concerns if a user has both the 
option to use a gateway allocation on a compute resource and also their own 
personal allocation (for example, when creating an experiment, does the user 
have two options for the compute resource, one with their own allocation and 
one with the gateway allocation?)



What we decided to do, at least for now, is to add a new role, similar to the 
gateway user role ("gateway-user"), that if a user has this new role then they 
can add their own compute/storage resource allocations. Also, if they have this 
new role they can only submit jobs to compute resource for which they have 
registered their own resource allocation.



I'm not quite sure what to call the new role. In the meeting we referred to 
this new role as a "campus user" role, since that is the use case we are 
targeting. That doesn't seem generic enough of a name, is there a better name 
to give to this role? I'm thinking about adding to pga_config.php:



  'personal-allocation-user-role-name' => 'campus-user'



I'm open to suggestions on the name of the role.



Thanks,



Marcus


​

RE: Building a Resource Allocation Service For Airavata

[Miller, Mark]

Another thing I did not mention, we provide users with an estimate of the 
amount of resource each job will use before they submit, and we provide a 
window in the interface that shows them what their consumption (current + 
running jobs) is.

Mark

From: Miller, Mark [mailto:mmil...@sdsc.edu]
Sent: Friday, September 23, 2016 12:39 PM
To: dev@airavata.apache.org
Subject: RE: Building a Resource Allocation Service For Airavata

Hi Anuj,

Just to follow up on Suresh’s comment.
We do not use an allocation protocol, though Sudhakar does, and probably can 
help with that.
We do enforce both storage and usage limits on users.

The former is done at login, and users over a certain threshold aren’t allowed 
to submit jobs until their storage levels are reduced.
The latter is enforced, but manually in our main site (at least at present).
However, our rest site has a better and more sophisticated method.

Our policy is to allow users to access 50,000 su if they are at a US 
institution, and 30,000 su if they are at a foreign institution.
When the user hits 10,20, 30, and 50 thousand SU, they receive an email to warn 
them about the process.
If they go above, I can turn off their account manually, but I usually ask them 
how they are doing first.

In the REST site, each time a user submits a job, we calculate the amount of 
total SUs it would consume if it ran to completion.
We subtract this amount from the amount of allocation the user has remaining. 
If the potential cost is greater than the amount they have remaining, the 
submission is rejected. That way, a user cannot submit 100 jobs each requiring 
6800 SU. Which is possible, though it would take a long time.

The idea of holding a reserve against potential expenses by a running job is 
not a new idea, but it prevents things from getting out of hand.

Let me know if you have any questions about how this works.

Best,
Mark



From: Suresh Marru [mailto:sma...@apache.org]
Sent: Friday, September 23, 2016 12:00 PM
To: Airavata Dev mailto:dev@airavata.apache.org>>
Subject: Re: Building a Resource Allocation Service For Airavata

Hi Anuj,

This is great summary. Since this is a new capability for Airavata, it will be 
useful to have dedicated wiki space to record all information. We can start 
with a high-level problem description (you have some of that in the form of use 
cases), architecture and design thoughts and Implementation plan (pointers to 
JIRA’s).

May be you can put it under - 
https://cwiki.apache.org/confluence/display/AIRAVATA/Architecture+Documentation

Mark and Terri might have some input on this task since CIPRES enforces soft 
quotas for users (but I guess there is no formal allocation request process). 
Once you have the wiki we can gather the feedback also there.

Thanks,
Suresh

On Sep 23, 2016, at 2:39 PM, Anuj Bhandar 
mailto:bhandar.a...@gmail.com>> wrote:

Hi Dev,

I have undertaken this project of designing and developing an resource 
allocation service for Airavata, to give you some context, this service takes 
inspiration from XRAS (XSEDE Resource Allocation Service).

Here is the preliminary design document, which was written as a part of 
GSOC-2016, 
https://cwiki.apache.org/confluence/display/AIRAVATA/%5BGsoc+Proposal%5D+Resource+Allocation+Manager+for+Apache+Airavata.

I have derived some use cases by discussing with Sudhakar Pamidighantam ( PI - 
GridChem), here is the link : 
https://issues.apache.org/jira/browse/AIRAVATA-2108.

Any comments and improvements are gladly welcome.

Thanks and best regards,

Anuj Bhandar

RE: Building a Resource Allocation Service For Airavata

[Miller, Mark]

Hi Anuj,

Just to follow up on Suresh’s comment.
We do not use an allocation protocol, though Sudhakar does, and probably can 
help with that.
We do enforce both storage and usage limits on users.

The former is done at login, and users over a certain threshold aren’t allowed 
to submit jobs until their storage levels are reduced.
The latter is enforced, but manually in our main site (at least at present).
However, our rest site has a better and more sophisticated method.

Our policy is to allow users to access 50,000 su if they are at a US 
institution, and 30,000 su if they are at a foreign institution.
When the user hits 10,20, 30, and 50 thousand SU, they receive an email to warn 
them about the process.
If they go above, I can turn off their account manually, but I usually ask them 
how they are doing first.

In the REST site, each time a user submits a job, we calculate the amount of 
total SUs it would consume if it ran to completion.
We subtract this amount from the amount of allocation the user has remaining. 
If the potential cost is greater than the amount they have remaining, the 
submission is rejected. That way, a user cannot submit 100 jobs each requiring 
6800 SU. Which is possible, though it would take a long time.

The idea of holding a reserve against potential expenses by a running job is 
not a new idea, but it prevents things from getting out of hand.

Let me know if you have any questions about how this works.

Best,
Mark



From: Suresh Marru [mailto:sma...@apache.org]
Sent: Friday, September 23, 2016 12:00 PM
To: Airavata Dev 
Subject: Re: Building a Resource Allocation Service For Airavata

Hi Anuj,

This is great summary. Since this is a new capability for Airavata, it will be 
useful to have dedicated wiki space to record all information. We can start 
with a high-level problem description (you have some of that in the form of use 
cases), architecture and design thoughts and Implementation plan (pointers to 
JIRA’s).

May be you can put it under - 
https://cwiki.apache.org/confluence/display/AIRAVATA/Architecture+Documentation

Mark and Terri might have some input on this task since CIPRES enforces soft 
quotas for users (but I guess there is no formal allocation request process). 
Once you have the wiki we can gather the feedback also there.

Thanks,
Suresh

On Sep 23, 2016, at 2:39 PM, Anuj Bhandar 
mailto:bhandar.a...@gmail.com>> wrote:

Hi Dev,

I have undertaken this project of designing and developing an resource 
allocation service for Airavata, to give you some context, this service takes 
inspiration from XRAS (XSEDE Resource Allocation Service).

Here is the preliminary design document, which was written as a part of 
GSOC-2016, 
https://cwiki.apache.org/confluence/display/AIRAVATA/%5BGsoc+Proposal%5D+Resource+Allocation+Manager+for+Apache+Airavata.

I have derived some use cases by discussing with Sudhakar Pamidighantam ( PI - 
GridChem), here is the link : 
https://issues.apache.org/jira/browse/AIRAVATA-2108.

Any comments and improvements are gladly welcome.

Thanks and best regards,

Anuj Bhandar

RE: [GSoC Proposal] In-Situ Simulation Monitoring and Analysis Using Apache Airavata

[Miller, Mark]

I like this proposal also very much, and agree with what Sandy said as well.

Mark


From: Sandra Gesing [mailto:sandra.ges...@nd.edu]
Sent: Tuesday, March 22, 2016 6:12 PM
To: dev@airavata.apache.org
Subject: Re: [GSoC Proposal] In-Situ Simulation Monitoring and Analysis Using 
Apache Airavata

Hi Jeff,

the proposal looks already excellent. I think it would be beneficial to have 
the feature in Airavata that users can directly search for certain terms in a 
log file and intermediate results additionally to see the whole log file and 
intermediate values.
Thanks,
Sandra


Sandra Gesing
Research Assistant Professor, Department of Computer Science and Engineering
Computational Scientist, Center for Research Computing
University of Notre Dame
http://www3.nd.edu/~sgesing


On Tue, Mar 22, 2016 at 8:03 PM, Jeffery Kinnison 
mailto:jeffery.d.kinniso...@nd.edu>> wrote:
Hello Dev Team,
I have created a draft of my GSoC proposal and made it available as at the link 
below.

https://docs.google.com/document/d/16BCTV__tA9h2Pnx1annaBIJaU2Jmma1ayTHV6BNMzNA/edit?usp=sharing
I would appreciate any comments to improve this proposal.
Best,
Jeff Kinnison

RE: [DISCUSS] Accept GenApp into Apache Airavata

[Miller, Mark]

Hi all,
 
As part of project leadership, I want to acknowledge that I read this, and 
think it sounds interesting, but I am not enough of a developer to comment on 
the technical reasonability and strategic merits, and suitability as part of 
Airavata or an incubated separate project. I am happy to go with Suresh and 
Marlon's wisdom on this.

Mark

-Original Message-
From: Marlon Pierce [mailto:marpi...@iu.edu] 
Sent: Wednesday, March 25, 2015 8:30 AM
To: dev@airavata.apache.org
Subject: Re: [DISCUSS] Accept GenApp into Apache Airavata

I support this. We have the PGA now as a way to illustrate how to use the API, 
but GenApp should provide a way for us to help developers create more 
application-centric user interfaces.

Marlon

On 3/25/15 11:14 AM, Marru, Suresh wrote:
> Hi All,
>
> Since there is not much discussion on this thread, I will make it more 
> explicit on one option (which is Emre’s preference) and solicit further 
> discussion.
>
> How about Emre can bring in GenApp into Airavata, integrate it further into 
> Airavata and bootstrap the community here. Some of Airavata Community might 
> get interested in GenApp. Once there is a quorum and feel if GenApp goals are 
> diverging from Airavata (or if the community becomes distinct), then GenApp 
> could go through incubator and get on a path towards a Apache Top Level 
> Project (TLP).
>
> Either way, Emre has been guiding GSoC students, on Airavata Integration, I 
> suggest they join the dev list and write their proposals on this list.
>
> Thoughts, concerns?
>
> Cheers,
> Suresh
>
> P.S Chris, I am taking liberty to send your private list reply to this, 
> please excuse.
>
> On Mar 22, 2015, at 3:45 PM, Mattmann, Chris A (3980) 
>  wrote:
>
> Hi everyone! Welcome Emre and if there are any questions about 
> Incubation I’d be happy to help answer them. Suresh great work 
> bringing the conversation to the lists.
>
> ++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398) NASA Jet 
> Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattm...@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Associate Professor, Computer Science Department University of 
> Southern California, Los Angeles, CA 90089 USA
> ++
>
>
>> On Mar 19, 2015, at 11:46 AM, Marru, Suresh  wrote:
>>
>> Hi Emre,
>>
>> I firstly applaud the CCP-SAS team in having GenApp as an Open Source 
>> project with an interest to grow a community. Normally, Apache has not been 
>> encouraging “umbrella” projects where lot of sub-projects exist. So the 
>> suggest mechanism, will be to go through the Incubator Process and work 
>> towards a GenApp top level project. But, here is a debatable situation and 
>> there are multiple ways we can view this.
>>
>> * GenApp can be considered a downstream project of Airavata in which case it 
>> can be argued for a stand alone project.
>> * GenApp is consuming Airavata API’s and helping users build gateways based 
>> on their applications (the lab generated code you refer below). In this 
>> perspective, GenApp could rightly belong into Airavata itself and have its 
>> own product releases. This is very similar to how Airavata currently 
>> releases XBaya and very soon a PHP Gateway.
>>
>> Ofcourse, there is a pragmatic way to approach this and have you get started 
>> within Airavata and if we realize its significant to stand on its own (and 
>> you might have generated some developer/community interest by then), this 
>> can then spin off into an incubator project and eventually into a TLP.
>>
>> I am cc’ing Airavata Community for input. I would have suggested apache 
>> incubator general list, but that may be early to start with. We should have 
>> expert advice here (there are a lot of apache members on airavata pmc and we 
>> have Chris Mattmann who is member of the current board).
>>
>> Cheers,
>> Suresh
>>
>>> On Mar 19, 2015, at 9:23 AM, Emre Brookes  wrote:
>>>
>>> Dear Suresh Marru,
>>>
>>> I am writing this letter pursuant to your consideration of the 
>>> inclusion of "GenApp" as a sub-project of Apache Airavata.
>>>
>>> The GenApp framework is a new open framework generating code on a 
>>> set of scientific modules that is easily extensible to new environments.
>>> For example, one can take a set of module definitions and generate a 
>>> complete HTML5/PHP science gateway and a Qt4/GUI application on the 
>>> identical set of modules.  If a new technology comes along, the 
>>> framework can easily be extended to new "target languages" by 
>>> including appropriate code fragments without effecting the 
>>> underlying modules. One motivation for the development was based 
>>> upon observation of scientific lab generated code, which 

RE: Saving the content of the STDOUT and STDERR to database

[Miller, Mark]

this technique has worked well for cipres. We refer all of our files to 
pointers.
We also use a hash key so there are no stored duplicate files.

Both steps save costly database space. Paul Hoover has done this for us, and he 
is available should you need advice.

Mark

From: Pamidighantam, Sudhakar V [mailto:spami...@illinois.edu]
Sent: Friday, January 09, 2015 7:15 PM
To: dev@airavata.apache.org
Subject: Re: Saving the content of the STDOUT and STDERR to database

If the STDOUT becomes large it is not good to store in a database table. 
Perhaps the data base can contain a pointer(URI)  to the file.

Sudhakar.
On Jan 9, 2015, at 5:01 PM, Chathuri Wimalasena 
mailto:kamalas...@gmail.com>> wrote:


Hi All,

At the moment, we are saving the content of the STDOUT and STDERR in to 
DATA_TRANSFER_DETAIL table in the database and when retrieving the whole 
experiment object, DataTransferDetail object is also included. Since we no 
longer using wrapper scripts for different applications (instead using the 
module itself), now the most of the applications write the output to STDOUT. 
This output might contain special characters which might ruin the json response 
at the client side. We face this issue with Gamess.

Now we treat STDOUT and STDERR as normal outputs which will be available to 
users at the end of the experiment. Due to that IMO, we no longer need to save 
the content of those files to database. If we still want to save them for some 
other reason, we should save them as files, not the string content. This change 
will need some database table data type modifications.

Feel free to provide your input.

Thanks..
Chathuri

RE: "input" and "output" subdirs in working directories

[Miller, Mark]

I believe this would work. Perhaps others can correct if there are cases where 
it is wrong.

-Original Message-
From: Marlon Pierce [mailto:marpi...@iu.edu] 
Sent: Tuesday, December 23, 2014 12:02 PM
To: dev@airavata.apache.org
Subject: Re: "input" and "output" subdirs in working directories

So let's take $working_dir as the base and upload all inputs to 
$working_dir/input. Then in the job script we set the working directory to 
$working_dir/output and run the application.

Would this be reasonably guaranteed to put all outputs in the 
$working_dir/output directory, or are there codes for which this would not work?

Marlon

On 12/23/14, 2:54 PM, Miller, Mark wrote:
> We do use a flat system. However, to return files in a very controlled way, 
> since one has to know precisely the names of input and output files, and 
> maybe even restrict the user's ability to name them. The codes we use produce 
> files conditionally, depending on the command line, and so unless you know 
> the code super well, using explicit names can cause certain output files to 
> now be returned at all, an obvious bummer for user and developer alike. As a 
> result, I have elected in most cases to just return all files. This can be 
> confusing for users, and just binning input and output as coarse categories 
> is something I wish I could do currently.
>
> That was my motivation in responding
>
> Mark
>
>
>
> -Original Message-
> From: Marlon Pierce [mailto:marpi...@iu.edu]
> Sent: Tuesday, December 23, 2014 11:50 AM
> To: dev@airavata.apache.org
> Subject: Re: "input" and "output" subdirs in working directories
>
> Thanks, Mark. What's your approach in CIPRES?  Do you use a flat structure 
> for your working directories, or do you do something else?
>
> Marlon
>
> On 12/23/14, 2:45 PM, Miller, Mark wrote:
>> I am not sure of the reasoning behind the design, but off-hand it seems to 
>> me it would simplify the job of returning to "input" and "output" files as 
>> discrete entities, so if you don't know which files are which, you would 
>> have at least two coarse bins that can be returned under separate banners.
>>
>> Mark
>>
>>
>> -Original Message-
>> From: Marlon Pierce [mailto:marpi...@iu.edu]
>> Sent: Tuesday, December 23, 2014 11:01 AM
>> To: Airavata Dev
>> Subject: "input" and "output" subdirs in working directories
>>
>> When Airavata executes a remote command (launching a SLURM job, for 
>> example), it creates a working directory on the target machine's scratch 
>> space and two subdirectories, "input" and "output".  Is there a good reason 
>> for creating these two subdirectories?  Why not just do all the work in the 
>> top level of the working directory?  It seems unnecessary.
>> Also, I don't understand why these are in the GFAC module, as these should 
>> be constructed from Registry information.
>>
>> Below is background information.
>> --
>> Below is an example working directory.
>>
>> $ cd
>> /oasis/scratch/trestles/ogce/temp_project/gta-work-dirs/TEST_8b10aa04
>> -
>> 95c3-4695-af77-d3b3987c7ef9/
>> $ ls -tlr
>> total 20
>> drwxr-xr-x 2 ogce sds128 4096 Dec 23 07:17 output
>> -rw-r--r-- 1 ogce sds128  831 Dec 23 07:39 1203922204.pbs
>> -rw--- 1 ogce sds128   28 Dec 23 07:40 Gaussian.stdout
>> -rw--- 1 ogce sds128  663 Dec 23 07:40 Gaussian.stderr drwxr-xr-x
>> 2 ogce sds128 4096 Dec 23 07:47 input
>>
>> The names of these subdirectories are specified in Constants.java (as
>> OUTPUT_DATA_DIR_VAR_NAME and INPUT_DATA_DIR_VAR_NAME).   Below are the
>> files in the GFAC module that use these two constants.
>>
>> $ find ./modules/gfac -type f -exec grep -il "OUTPUT_DATA_DIR_VAR" {} 
>> \;
>> | grep java|grep -v target
>>
>> ./modules/gfac/gfac-core/src/main/java/org/apache/airavata/gfac/Const
>> a
>> nts.java
>> ./modules/gfac/gfac-core/src/main/java/org/apache/airavata/gfac/core/
>> c
>> pi/BetterGfacImpl.java
>> ./modules/gfac/gfac-gram/src/main/java/org/apache/airavata/gfac/gram/
>> u
>> til/GramRSLGenerator.java
>> ./modules/gfac/gfac-local/src/main/java/org/apache/airavata/gfac/loca
>> l
>> /provider/impl/LocalProvider.java
>> ./modules/gfac/gfac-ssh/src/main/java/org/apache/airavata/gfac/ssh/pr
>> o
>> vider/impl/SSHProvider.java
>>
>> So we would need to clean these up if we remove the constants.
>>
>> Marlon
>>

RE: "input" and "output" subdirs in working directories

[Miller, Mark]

We do use a flat system. However, to return files in a very controlled way, 
since one has to know precisely the names of input and output files, and maybe 
even restrict the user's ability to name them. The codes we use produce files 
conditionally, depending on the command line, and so unless you know the code 
super well, using explicit names can cause certain output files to now be 
returned at all, an obvious bummer for user and developer alike. As a result, I 
have elected in most cases to just return all files. This can be confusing for 
users, and just binning input and output as coarse categories is something I 
wish I could do currently.

That was my motivation in responding

Mark



-Original Message-
From: Marlon Pierce [mailto:marpi...@iu.edu] 
Sent: Tuesday, December 23, 2014 11:50 AM
To: dev@airavata.apache.org
Subject: Re: "input" and "output" subdirs in working directories

Thanks, Mark. What's your approach in CIPRES?  Do you use a flat structure for 
your working directories, or do you do something else?

Marlon

On 12/23/14, 2:45 PM, Miller, Mark wrote:
> I am not sure of the reasoning behind the design, but off-hand it seems to me 
> it would simplify the job of returning to "input" and "output" files as 
> discrete entities, so if you don't know which files are which, you would have 
> at least two coarse bins that can be returned under separate banners.
>
> Mark
>
>
> -Original Message-
> From: Marlon Pierce [mailto:marpi...@iu.edu]
> Sent: Tuesday, December 23, 2014 11:01 AM
> To: Airavata Dev
> Subject: "input" and "output" subdirs in working directories
>
> When Airavata executes a remote command (launching a SLURM job, for example), 
> it creates a working directory on the target machine's scratch space and two 
> subdirectories, "input" and "output".  Is there a good reason for creating 
> these two subdirectories?  Why not just do all the work in the top level of 
> the working directory?  It seems unnecessary.
> Also, I don't understand why these are in the GFAC module, as these should be 
> constructed from Registry information.
>
> Below is background information.
> --
> Below is an example working directory.
>
> $ cd
> /oasis/scratch/trestles/ogce/temp_project/gta-work-dirs/TEST_8b10aa04-
> 95c3-4695-af77-d3b3987c7ef9/
> $ ls -tlr
> total 20
> drwxr-xr-x 2 ogce sds128 4096 Dec 23 07:17 output
> -rw-r--r-- 1 ogce sds128  831 Dec 23 07:39 1203922204.pbs
> -rw--- 1 ogce sds128   28 Dec 23 07:40 Gaussian.stdout
> -rw--- 1 ogce sds128  663 Dec 23 07:40 Gaussian.stderr drwxr-xr-x 
> 2 ogce sds128 4096 Dec 23 07:47 input
>
> The names of these subdirectories are specified in Constants.java (as
> OUTPUT_DATA_DIR_VAR_NAME and INPUT_DATA_DIR_VAR_NAME).   Below are the
> files in the GFAC module that use these two constants.
>
> $ find ./modules/gfac -type f -exec grep -il "OUTPUT_DATA_DIR_VAR" {} 
> \;
> | grep java|grep -v target
>
> ./modules/gfac/gfac-core/src/main/java/org/apache/airavata/gfac/Consta
> nts.java 
> ./modules/gfac/gfac-core/src/main/java/org/apache/airavata/gfac/core/c
> pi/BetterGfacImpl.java 
> ./modules/gfac/gfac-gram/src/main/java/org/apache/airavata/gfac/gram/u
> til/GramRSLGenerator.java 
> ./modules/gfac/gfac-local/src/main/java/org/apache/airavata/gfac/local
> /provider/impl/LocalProvider.java 
> ./modules/gfac/gfac-ssh/src/main/java/org/apache/airavata/gfac/ssh/pro
> vider/impl/SSHProvider.java
>
> So we would need to clean these up if we remove the constants.
>
> Marlon
>

RE: "input" and "output" subdirs in working directories

[Miller, Mark]

I am not sure of the reasoning behind the design, but off-hand it seems to me 
it would simplify the job of returning to "input" and "output" files as 
discrete entities, so if you don't know which files are which, you would have 
at least two coarse bins that can be returned under separate banners.

Mark


-Original Message-
From: Marlon Pierce [mailto:marpi...@iu.edu] 
Sent: Tuesday, December 23, 2014 11:01 AM
To: Airavata Dev
Subject: "input" and "output" subdirs in working directories

When Airavata executes a remote command (launching a SLURM job, for example), 
it creates a working directory on the target machine's scratch space and two 
subdirectories, "input" and "output".  Is there a good reason for creating 
these two subdirectories?  Why not just do all the work in the top level of the 
working directory?  It seems unnecessary.  
Also, I don't understand why these are in the GFAC module, as these should be 
constructed from Registry information.

Below is background information.
--
Below is an example working directory.

$ cd
/oasis/scratch/trestles/ogce/temp_project/gta-work-dirs/TEST_8b10aa04-95c3-4695-af77-d3b3987c7ef9/
$ ls -tlr
total 20
drwxr-xr-x 2 ogce sds128 4096 Dec 23 07:17 output
-rw-r--r-- 1 ogce sds128  831 Dec 23 07:39 1203922204.pbs
-rw--- 1 ogce sds128   28 Dec 23 07:40 Gaussian.stdout
-rw--- 1 ogce sds128  663 Dec 23 07:40 Gaussian.stderr drwxr-xr-x 2 ogce 
sds128 4096 Dec 23 07:47 input

The names of these subdirectories are specified in Constants.java (as 
OUTPUT_DATA_DIR_VAR_NAME and INPUT_DATA_DIR_VAR_NAME).   Below are the 
files in the GFAC module that use these two constants.

$ find ./modules/gfac -type f -exec grep -il "OUTPUT_DATA_DIR_VAR" {} \; 
| grep java|grep -v target

./modules/gfac/gfac-core/src/main/java/org/apache/airavata/gfac/Constants.java
./modules/gfac/gfac-core/src/main/java/org/apache/airavata/gfac/core/cpi/BetterGfacImpl.java
./modules/gfac/gfac-gram/src/main/java/org/apache/airavata/gfac/gram/util/GramRSLGenerator.java
./modules/gfac/gfac-local/src/main/java/org/apache/airavata/gfac/local/provider/impl/LocalProvider.java
./modules/gfac/gfac-ssh/src/main/java/org/apache/airavata/gfac/ssh/provider/impl/SSHProvider.java

So we would need to clean these up if we remove the constants.

Marlon

RE: Partial results of an application run

[Miller, Mark]

Aha, Very interesting, I was not thinking about the full span of possibilities. 
You make a good point.
In fact, CIPRES has a special system that manages files above a certain size, 
because otherwise they just gum up our whole app and bring the server down.
These are regular result files.

I agree there may be cases where the Gateway needs/wants  to refuse certain 
files, such as the example you mentioned. 
For us it possible, but a lot more work, to specify each file that needs to be 
returned. But for apps with one or two codes that are well understood it would 
be very little effort.

I am wondering if the large files you mentioned are produced by gadget only in 
the failure case?
And how will/does  Airavata manage files that are very large in the success 
case? 
And how large of files can we manage in Airavata?

But now that's a whole new thread I fear


Mark


-Original Message-
From: Raminder Singh [mailto:raminderjsi...@gmail.com] 
Sent: Friday, September 19, 2014 2:28 PM
To: dev@airavata.apache.org
Subject: Re: Partial results of an application run

Thanks Mark for the feedback. I agree with you that SciGap should provide all 
the information to the gateway to debug a job. Followup question is, we should 
make this a default behavior or gateway configured behavior? I had worked with 
codes like Gadget in the past which produce huge intermediate files. In that 
case, we don't want to transfer all the files. I would suggest gateway to 
provide details about the file it needs to debug a job failure, if not the real 
intermediate output name some regular expression to find them and transfer to 
the gateway server.  

Other way of dealing with this can be transfer intermediate files on demand. 
Gateway admins or users get the list of intermediate files in the working 
folder. Airavata only transfer files based on the user selection. Just an idea. 

Thanks
Raminder

On Sep 19, 2014, at 12:18 PM, Marlon Pierce  wrote:

> Great feedback, Mark--
> 
> Marlon
> 
> On 9/19/14, 12:03 PM, Miller, Mark wrote:
>> Hi Raminder,
>> 
>> If I understand the issue, I have a comment.
>> 
>> The stdout/stderr files are absolutely critical to a gateway runner and the 
>> every end user for debugging issues, even simple ones like a typo that 
>> breaks the infile formatting.
>> There are two levels of benefit: first, the savvy user reads these docs and 
>> solves their own problem more quickly, if it is on their side, and notifies 
>> me of the message when it isn't on their side. This saves time for everyone.
>> Second, when the non-savvy user reports an issue but can't figure out what's 
>> wrong,, this is the first place I look to identify the issue. It also makes 
>> it possible at all to debug on a reasonably large user population. removing 
>> these files takes away the levers both users and gateway owners have to 
>> manage issues. If I understand the issue correctly, I don't think that 
>> SciGap wants to inherit the job of debugging job runs for all its 
>> constituent Gateways, and not passing these files along would do just that.
>> I strongly feel, again assuming I understand correctly, that all the files 
>> available from every failed job should be passed along to the Gateway by 
>> SciGap. If the Gateway owners wish to debug every user issue on their own, 
>> they can pass only certain files along to the user.
>> 
>> In our time with CIPRES I think I have used almost every file we or the 
>> users job has created to debug an issue at one time or another.
>> Sometimes the absence of a file alone tells me what the issue is. On the 
>> other hand, many of code produce both STDOUT as a file, and stdout.txt as a 
>> file.
>> If SciGap wanted to be responsible for eliminating that ambiguity, 
>> that might be fine. But delivering both copies puts the effort back on the 
>> Gateway developer to decide how to handle it, and perhaps that is again the 
>> best solution. And it requires no effort on the part of SciGap, which 
>> already has many things to take care of.
>> 
>> So my vote is to return everything to the Gateway for every job.
>> 
>> The exception with this might be SciGap-created files that do not have any 
>> relevance to the Gateway. I still think they would be of benefit to the 
>> Gateway developer (at least) because then they can report the issue to 
>> SciGap directly and explicitly.
>> That means the admin on the SciGap side does not have to look up the job, 
>> maneuver to the directory, and open files to find the error message. It 
>> conserves many keystrokes/clicks. If we have 50 client Gateways, we will be 
>> grateful when a User reports the SciGap error message and job number, 

RE: Evaluate Suitable Scientific Workflow Language for Airavata.

[Miller, Mark]

Very interesting thoughts Bruce.
The issue of sustainability is certainly a key one, and building dependencies 
on another platform always brings risk.

Going back to Lavanya’s comment, I think this thought is critical.
What workflow tools in use really have big uptake by users?

At CIPRES we have tended to focus on being driven by what users are actually 
asking for
(of course we are always behind schedule, but in adding new features this is 
not the worst thing).
What do we know about the requirements for workflow within Airavata from the 
user POV?

It seems like the correct place to start.

Mark

From: Bruce Barkstrom [mailto:brbarkst...@gmail.com]
Sent: Friday, September 19, 2014 9:29 AM
To: d...@oodt.apache.org
Cc: architect...@airavata.apache.org; dev
Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.

One factor that should be included in the group's deliberations
on adding a workflow language to the other things in OODT
is the impact on long-term maintenance.  While there's a lot
of enthusiasm in the developer community right now, we need
to think about what happens when development turns into
maintenance.  The account that follows is based on my experience
with trying to resurrect a W3C-related project to visualize RDF
graphs.
The project is called IsaViz.  It's even got a W3 web site:
http://www.w3.org/2001/11/IsaViz/
IsaViz identifies itself as a visual authoring tool for RDF.
Right up near the top are two dates that should serve as
a cautionary note for people who want to pick up this tool:
Current Stable Version: October 2007 and Current Development Version:
May 2007.  It also looks like the site was maintained by a single
developer who did the development as a postdoc project.
The overview page was last modified on Oct. 21, 2007.
The installation instructions were last modified on Oct. 18, 2004.
IsaViz uses a number of tools.  The Installation Instructions
identify the following:

  *   A Java Virtual Machine version 1.3.0 or later (1.4 strongly recommended - 
see Known problems)
  *   A distribution of IsaViz, which contains the following Java JAR files:

 *   IsaViz itself (isaviz.jar)
 *   Zoomable Visual Transformation Machine (zvtm.jar)
 *   Jena 2.1 for IsaViz 2.1, Jena 1.6.1 for IsaViz 1.2
 *   Xerces-J version 2 (xercesImpl.jar,xmlParserAPIs.jar) for IsaViz 2, 
Xerces 1.4.4 for IsaViz 1.2

  *   GraphViz from AT&T version 1.8.9 or later (version 1.7.x is no longer 
supported in IsaViz 2, and has actually only been tested with version 1.9). 
Note: some instances of version 1.10.0 had a bug that produced incomplete SVG 
files, but it has been corrected in subsequent releases (newer versions can be 
obtained on the graphviz.org 
site).
So, what complications ensue:
1.  Java has moved way beyond version 1.3 or 1.4.  Since Java can deprecate 
code and
since there's Oracle and OpenJDK, there may be some unpleasantries that might 
need
fixes.  I haven't seen comments from the community on whether or not these 
might be
significant.  The IsaViz documentation refers to the ancient time when Sun 
controlled
the language.  Apparently, the IsaViz code was only tested with Sun's j2se/1.3 
or 1.4.
2.  I suppose the jar files from IsaViz version 2 would be the place to start 
in reconstructing
this piece of software.  However, one might be careful about this when you get 
into the
installation scripts from the Installation Instructions.
3.  The Zoomable Visual Transformation Machine project is on Sourceforge.  It's 
apparently
done in Java.  However, the IsaViz code used version 0.9.0, while the current 
Sourceforge
project (at http://zvtm.sourceforge.net/) is now up to 0.11.1 for the stable 
version (Aug. 2013)
with a more recent development version (0.11.2 - snapshot; June 2014).  No idea 
if
there would be any serious ramifications from this change.
4.  The Installation Instructions have a link to the HP Jena site.  If you link 
to it, the
page says "Oops! ..."  Jena was moved from HP to apache.  So if you want to do
Jena, you now need to consult .  I'm not sure exactly 
how
the apache Jena source code or binary installations compare with what IsaViz is 
expecting.
As a note, Jena is a BIG chunk of software.  I think the tutorials on RDF 
(including OWL
and related reasoners) are going to take a novice user (including many 
scientists) a month
or two of dedicated time to work through.  I don't know how easy IsaViz would 
be to install
without at least a basic understanding of RDF and of the related triple store 
database.
5.  Xerces-J is the XML Java parser (see 
http://Xerces.apache.org>>), which is now up to
version 2.11.0.  Again, it isn't clear what kinds of difficulties one would 
encounter to use
this library.
6.  GraphViz (at ) is now at version 2.38.0-1.  AT&T 
seems
to be maintaining a lot of install

RE: Partial results of an application run

[Miller, Mark]

Hi Raminder,

If I understand the issue, I have a comment.

The stdout/stderr files are absolutely critical to a gateway runner and the 
every end user for debugging issues, even simple ones like a typo that breaks 
the infile formatting.
There are two levels of benefit: first, the savvy user reads these docs and 
solves their own problem more quickly, if it is on their side, and notifies me 
of the message when it isn't on their side. This saves time for everyone.
Second, when the non-savvy user reports an issue but can't figure out what's 
wrong,, this is the first place I look to identify the issue. It also makes it 
possible at all to debug on a reasonably large user population. removing these 
files takes away the levers both users and gateway owners have to manage 
issues. If I understand the issue correctly, I don't think that SciGap wants to 
inherit the job of debugging job runs for all its constituent Gateways, and not 
passing these files along would do just that.
I strongly feel, again assuming I understand correctly, that all the files 
available from every failed job should be passed along to the Gateway by 
SciGap. If the Gateway owners wish to debug every user issue on their own, they 
can pass only certain files along to the user.

In our time with CIPRES I think I have used almost every file we or the users 
job has created to debug an issue at one time or another.
Sometimes the absence of a file alone tells me what the issue is. On the other 
hand, many of code produce both STDOUT as a file, and stdout.txt as a file.
If SciGap wanted to be responsible for eliminating that ambiguity, that might 
be fine. But delivering both copies puts the effort back on the Gateway 
developer to decide how to handle it, and perhaps that is again
the best solution. And it requires no effort on the part of SciGap, which 
already has many things to take care of.

So my vote is to return everything to the Gateway for every job.

The exception with this might be SciGap-created files that do not have any 
relevance to the Gateway. I still think they would be of benefit to the Gateway 
developer (at least) because then they can report the issue to SciGap directly 
and explicitly.
That means the admin on the SciGap side does not have to look up the job, 
maneuver to the directory, and open files to find the error message. It 
conserves many keystrokes/clicks. If we have 50 client Gateways, we will be 
grateful when a User reports the SciGap error message and job number, rather 
than just saying they have an issue.

Those are my thoughts.

Best,
Mark


From: Raminder Singh [mailto:raminderjsi...@gmail.com]
Sent: Friday, September 19, 2014 7:56 AM
To: dev@airavata.apache.org
Subject: Partial results of an application run

Hi Dev,

Currently we are not moving partial results (stdout/stderr and other files) to 
the gateway incase of application failure (failed to produce output). This can 
be fixed but question is do we want it to be the default behavior or based on 
some user flag in experiment. To make it work properly, we need user input to 
provide regular expressions or other details about required files, incase of 
failure. Any suggestions on these changes in Application catalog and Airavata 
API. We also need APIs functions to get job working directory and other 
details. I created a JIRA for this.

https://issues.apache.org/jira/browse/AIRAVATA-1449

Thanks
Raminder

RE: Experiment log messages to different log files

[Miller, Mark]

Re max files, here is what I found:

It depends on the file system. ext3 suppport ~32000 subdirectories (not files!) 
in a given directory, with ext4 it's 64000 by default. xfs has no limit to my 
knowledge.  You should consider not putting too many files in a single 
directory. Most software doesn't handle that well (e.g. mc will be slow, many 
gui tools will be unusable). It's better to create a hierarchy of nested 
folders and distribute the files in them using some algorithm (hash of file 
name or content or any other method which will distribute the files equally). 
That's what many mature programs (e.g. squid) do. 


-Original Message-
From: Marlon Pierce [mailto:marpi...@iu.edu] 
Sent: Thursday, August 28, 2014 7:54 AM
To: Airavata Dev
Subject: Experiment log messages to different log files

I'm wondering if it would be a good idea to create a separate log file for each 
experiment.  We current direct everything to airavata.log and 
airavata-server.out. As a consequence, log entries for different submissions 
get interleaved, which makes tracing a particular experiment's life cycle 
difficult.

Note there is a 32,000 file limit per directory in Linux by default (unless my 
knowledge is obsolete), so we'd need to take this into account.

Alternatively, we could do a better job of labeling the entries in airavata.log 
so that it was clear which experiment is associated with the entry.

Other suggestions?

Marlon

RE: Experiment Cancellation

[Miller, Mark]

If I understand this correctly, I want to offer some input from our experience 
with CIPRES.
Currently, if a CIPRES user wishes to cancel a job, they must delete the entire 
job, and therefore all ability to view the input and other files used become 
unavailable.
This is not an ideal solution.

There is value to the user to being able to see partially completed results, or 
even the input files they used.

So I would vote for making partial output of the job available as an option.
Any additional information you can provide about status would be useful, 
especially for folks who are debugging failures..

Just my 2c.

Mark

From: Eroma Abeysinghe [mailto:eroma.abeysin...@gmail.com]
Sent: Wednesday, August 13, 2014 7:04 AM
To: dev@airavata.apache.org
Subject: Re: Experiment Cancellation

My questions and thoughts on Experiment cancellation
1. What are we going to do for output or partial output of the job at the time 
of cancelling?
Are we going to discard or make them available for the experiment. Are we 
safe keeping all the job information, messages on CANCELLED jobs or discard 
them as well?

2. Are we going to allow editing for CANCELLED or CANCELLING experiments?
IMO we should not. because allowing editing is required if its going to 
Re-launch.

3. With existing experiment and job states we need to decide which are going to 
be CANCELLED
Out of Airavata Experiment states Cancellation should be allowed for states;
CREATED
VALIDATED
SCHEDULED
LAUNCHED
EXECUTING
Cancellation should be communicated to resources if the job states are;
SUBMITTED
SETUP
QUEUED
ACTIVE
HELD


There is SUSPENDED state in both experiment and job but is this a currently 
active state?

4. Cloning will be available for CANCELLED and CANCELLING experiments.

5. In Experiment Summary we should display any errors took place in cancelling 
process












On Wed, Aug 13, 2014 at 9:01 AM, Marlon Pierce 
mailto:marpi...@iu.edu>> wrote:
There is an advantage for task (or job) state to capture the information that 
really comes from the machine (completed, cancelled, failed, etc), and for 
experiment state to be set to canceled by Airavata.  That is, there should be 
parts of Airavata that capture machine-specific state information about the job 
for logging/auditing purposes.

* Airavata issues "cancel" command to job in "launched" or "executing" state.

* Airavata confirms that the job has left the queue or is no longer executing. 
This could be machine-specific, but the main question is "has the job left the 
queue?" or "is the job no longer in executing state?"  I don't think it is "if 
this is trestles, and since we issued a qdel command, is the job marked as 
completed; of if this is stampede, is the job now marked as failed?"

* If the job cancel works, the Airavata marks this as canceled.

* If cancel fails for some reason, don't change the Experiment state but throw 
an error.


Marlon


On 8/13/14, 2:57 AM, Lahiru Gunathilake wrote:
Hi All,

I have few concerns about experiment cancellation. When we want to cancel
and experiment we have to run a particular command in the computing
resource. Based on the computing resource different resources show the job
status of the cancelled jobs in a different way. Ex: trestles shows the
cancelled jobs as completed, some other machines show it as as cancelled,
some might show it as failed.

I think we should replicated this information in the JobDetails object as
the Job status and make sure the Experiments and Task statuses as
cancelled. The other approach is when we cancel we explicitly make all the
states in the experiment model (experiments,tasks,job states as cancelled)
as cancelled and manually handle the state we get from the computing
resource.

My concerns should we really hide that information shown in the computing
resource from the Job status we are storing in to the registry ? or leave
it as it is and handle other statuses to represent the cancelled
experiments ? If we make everything cancel there will be inconsistency in
the JobStatus.

WDYT ?

Lahiru




--
Thank You,
Best Regards,
Eroma

RE: [SciGaP-Dev] CIPRES-Airavata integration proposal]]

[Miller, Mark]

I wanted to chime in and say,  that's really the point of Kenneth's activity.
We do not expect you all to be able to provide everything that a new developer 
will need to know. I find it super-efficient to make a credible first pass, and 
assume there will be holes. He is a test case to drive requirements and help 
create documentation for all who follow

Mark

-Original Message-
From: Marlon Pierce [mailto:marpi...@iu.edu] 
Sent: Thursday, July 24, 2014 12:50 PM
To: Yoshimoto, Kenneth; dev@airavata.apache.org
Cc: d...@scigap.org
Subject: Re: [SciGaP-Dev] CIPRES-Airavata integration proposal]]

It's in airavata-api/thrift-interface-descriptions/ in your checkout, and in 
https://github.com/apache/airavata/tree/master/airavata-api/thrift-interface-descriptions
on github.

"You should dig around in the API to solve your problems" is not my preferred 
response.  We just don't have all the examples we need. Your feedback and 
contributions of use cases and scripts will help us pull these together.

Marlon

On 7/24/14, 3:44 PM, K Yoshimoto wrote:
> Thanks, Marlon.  Where do I find experimentModel.thrift?
>
> On Thu, Jul 24, 2014 at 03:36:49PM -0400, Marlon Pierce wrote:
>> Note in general that the PHP scripts are just illustrative and don't 
>> show everything.  However, you do have access to everything in the 
>> API in these scripts if you modify them.
>>
>> Maybe the best way to see what can be done is to look at the Thrift 
>> models. In this case, the experimentModel.thrift has the complete 
>> description.  These are all programmatically available in PHP.
>>
>>
>> Marlon
>>
>>
>> On 7/24/14, 3:31 PM, Singh, Raminder jeet wrote:
>>> Kenneth,
>>>
>>> It looks like your application id is NULL.  Can you please look into 
>>> your sample experiment create/launch code? Ideally experiment 
>>> validator should though such exception to the user if application id 
>>> is not set. I will look into the validation code. I am working to 
>>> get experiment errors using a function call.
>>>
 php getExperiment.php pwdtest_e08d60fb-62e9-4d54-bfe4-8688d894585d
>>> object(Airavata\Model\Workspace\Experiment\Experiment)#6 (18) {
>>>["experimentID"]=>
>>>string(44) "pwdtest_e08d60fb-62e9-4d54-bfe4-8688d894585d"
>>>["projectID"]=>
>>>string(48) "kennethtest_deebd5c1-2b77-481b-a01c-9c8213cdc560"
>>>["creationTime"]=>
>>>int(1406155998504)
>>>["userName"]=>
>>>string(7) "kenneth"
>>>["name"]=>
>>>string(7) "pwdtest"
>>>["description"]=>
>>>NULL
>>>["applicationId"]=>
>>>NULL
>>>["applicationVersion"]=>
>>>NULL
>>>
>>> Thanks
>>> Raminder
>>>
>>>
>>> On Jul 24, 2014, at 2:36 PM, K Yoshimoto  wrote:
>>>
 I took a look through:
 RegisterSampleApplications.java  
 RegisterSampleApplicationsUtils.java

 I see the examples for setting up the various apps and compute hosts.

 some questions:
 - is there an equivalent php sample set for doing this?
 - what's the easiest way to build the java samples, if those have
   more example coverage than the php samples?
 - is there a document for supported strings for resourceJobManagerType?

 On Wed, Jul 23, 2014 at 10:45:12PM -0400, Lahiru Gunathilake wrote:
> Did you have a look in to RegisterSampleApplications.java class ?
>
> registerXSEDEHosts method is registering bigred2.uits.iu.edu and 
> its an ssh host, but in your case your job manager type will be FORK.
>
> If you can connect to this head node using ssh protocol airavata 
> should work if its gsissh then we do not support it at this point. 
> Configure your ssh key configurations in 
> airavata-server.properties (either username password or keys) or 
> You can store these information to the credential store and parse 
> the tokenId generated during credential storing and use that token 
> along the request.
>
> # -- For ssh key pair authentication
> ---  #public.ssh.key=/path to 
> public key for ssh #ssh.username=username for ssh connection 
> #private.ssh.key=/path to private key file for ssh 
> #ssh.keypass=passphrase for the private key
>
>
> # -- For ssh key pair authentication
> ---  #ssh.username=username for 
> ssh connection #ssh.password=Password for ssh connection
>
> I will try to do the same thing in stampede and get back to you.
>
> Regards
> Lahiru
>
>
>
>
> On Jul 23, 2014, at 7:05 PM, K Yoshimoto wrote:
>
>> Great info!  How do I register a host as a non-hpc resource?
>> I didn't see an obvious variable name in the 
>> registerComputeResource.php example script.
>>
>> On Wed, Jul 23, 2014 at 01:56:51PM -0400, Lahiru Gunathilake wrote:
>>> On Jul 23, 2014, at 1:49 PM, K Yoshimoto wrote:
>>>
 Hi, just followin

RE: PHP-Reference-Gateway | Query

[Miller, Mark]

As a user, I like the null string option and that behavior for when I cant 
remember exactly what the name of something I am looking for is.
It’s not tidy, but it keeps me from having to pull out something else and look 
for it, if I recognize it when I see it.
So I see a big plus to this personally, as a user. If I have too many files, I 
use a wild card to make the list smaller.

I think having both, the former for disorganized users like me, and the latter 
“browse” for those who are more orderly.

Just my 2c

Mark

From: Reagan, David Michael [mailto:dmrea...@iu.edu]
Sent: Wednesday, June 04, 2014 1:45 PM
To: dev@airavata.apache.org
Subject: RE: PHP-Reference-Gateway | Query

Hi, Gagan. I’m not sure how the API handles a null search query, but I 
purposefully chose to make the field mandatory in the gateway. Maybe it’s just 
me, but I find it strange that it would return everything when you search for 
nothing.

If it behaved as you describe, the page would list all of the 
experiments/projects owned by that user. Presently, that functionality is 
performed by the Browse page. Perhaps it would make sense to eliminate the 
Browse page, and just use search for everything. I would be OK with that, but I 
would advocate for the user having to enter a wildcard character like ‘*’ 
instead of accepting an empty string.

Dave

From: Gagan Juneja [mailto:gagandeepjun...@gmail.com]
Sent: Monday, June 02, 2014 1:41 PM
To: dev@airavata.apache.org
Subject: PHP-Reference-Gateway | Query

Hi Team,
I am playing with the PHP version deployed of Airavata API 
(http://gw120.iu.xsede.org/PHP-Reference-Gateway/search_projects.php). While 
doing search for an experiment or a project the search keyword (based on the 
criteria I selected) is mandatory field. Which In my opinion should not be 
mandatory and should return all possible results for that category.

Is there any specific reason behind making this field mandatory?


Regards,
Gagan

RE: Validation failures and storing the failures

[Miller, Mark]

I feel as long as they are stored somewhere, that should be fine.
The question is, if Airavata doesn't have access to the gateway database, will 
they be hamstrung by the inability to retrieve failure messages that might be 
informative?
Maybe everything will be in the logs anyhow?

Mark

-Original Message-
From: Marlon Pierce [mailto:marpi...@iu.edu] 
Sent: Wednesday, May 14, 2014 12:23 PM
To: dev@airavata.apache.org
Subject: Re: Validation failures and storing the failures

I don't think these need to be stored, just sent back to the gateway. 
If I recall correctly, steps are

1. Gateway invokes "execute" method of API server.

2. API Server runs Orchestrator's "validate()".

3. Orch returns result of validation.

4. API server returns results to the gateway.

5. If validation passed, API Server then runs the Orchestrator's "launch()".

So step #4 should enough.

Marlon


On 5/14/14 12:17 PM, Lahiru Gunathilake wrote:
> Hi All,
>
> We recently added validation logics to orchestrator, so gateway 
> developers can add/configure their validators to be invoked. If they 
> failed they can wrap an error message in their validation logic and return 
> that.
> Orchestrator needs to take these error messages and store them to a 
> persistent storage so that if the validation failed these errors can 
> be showed to the gateway user to make it correct. Unless we give a 
> proper error message to the end user there is no way to make it correct.
>
> Currently in the orchestrator implementation it does not store the 
> error but we have a proper result object which keeps the validation 
> state and a message.
>
> Where should I save this information in our data model ? This is an 
> information specific to each invocation and there could be multiple 
> error messages came from multiple validators because we do not return 
> immediately after the first validation failure but rather we invoke 
> all the validators and collect all the errors (currently we just have 
> access to them and simply logging those errors in server side). If 
> there is no placeholder for these validation errors can we add 
> something to the data model to store them and retrieve them from the client 
> code ?
>
> I have added FAQ type of a document for Orchestrator[1], please 
> provide your feedback on this.
>
> [1]
> https://docs.google.com/document/d/1FOc0X6HCMZ9E-fTnZQ8aGo7tK20C8lvKly
> cNouswICI/edit
>
> Regards
> Lahiru
>

RE: Orchestrator description draft

[Miller, Mark]

Thanks for doing that Marlon,

I have a couple of questions, sorry for my naivete.

Is the term CPI specific to Airavata? I have not heard it before.

When you state:
"Note jobs can complete but tasks can fail."

do you mean:
"Note that although jobs within a given task can complete,  the task they are 
contained in may fail."?

Question:
Is it really possible for a user-initiated activity to be persistent? Maybe I 
don't understand the use case or language.
I wonder how scalable it can be for user-initiated activity to persist.

Question:
If jobs can be persistent, tasks and workflows may also be persistent, right? 
This also seems potentially like an issue, if my understanding of persistence 
is correct.

For the Orchestrator:
If it knows the status of XSEDE and other resources, do we know how it gets 
that information? Is there a specific way it plugs in to other remote resources 
that ensures that info is provided (in other words, there are many kinds of 
resources, and perhaps many ways of broadcasting their condition; or maybe it 
is just online/offline?)

Also, if Orchestrator knows the status of the remote resources, can it pass 
that information forward to the Gateway front end, so it can be printed in the 
user interface somewhere? From my perspective, it is way cooler if the user 
knows before submitting that there will be a delay, or re-routing of their job.

Mark

-Original Message-
From: Marlon Pierce [mailto:marpi...@iu.edu] 
Sent: Wednesday, May 14, 2014 8:02 AM
To: dev@airavata.apache.org; architect...@airavata.apache.org
Subject: Orchestrator description draft

Dear all--

I've written up a draft description of the Orchestrator [1] and welcome 
comments and critiques.  As with the GFAC description, this is not necessarily 
based on the current implementation.  The purpose is to create an 
implementation-independent description of the Orchestrator for future reference.

Some outcomes from this exercise:

* The interactions of the Workflow Interpreter, Orchestrator, and API server 
need to be thought out. Don't take my suggestions here too seriously.

* The scheduler component of the Orchestrator needs more thought, especially if 
there are multiple Orchestrators running (for load
balancing): we don't want to run into "thread" issues if multiple schedulers 
are trying to work with the registry.

* Our current concept for extending the Orchestrator is to extend the CPI.  You 
would do this to implement, for example, more sophisticated scheduling.  But we 
could take a GFAC approach of having a core and developer-provided plugins (for 
scheduling, quality of service, etc).

Marlon

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=40511565

RE: Validation failures and storing the failures

[Miller, Mark]

Remember, I am on the naïve side so I am not saying what should be in terms of 
implementation.
Certainly, passing to the Gateway for storage in the Gateway database is a good 
solution for the Gateway.

I was wondering if the Airavata operators might need access to the same 
information, for debugging purposes. 

-Original Message-
From: Marlon Pierce [mailto:marpi...@iu.edu] 
Sent: Wednesday, May 14, 2014 12:33 PM
To: dev@airavata.apache.org
Subject: Re: Validation failures and storing the failures

I was assuming the gateway would figure out what to do with the errors after 
step 4.  This obviously needs more thought on how this would work.

Marlon

On 5/14/14 3:27 PM, Miller, Mark wrote:
> I feel as long as they are stored somewhere, that should be fine.
> The question is, if Airavata doesn't have access to the gateway database, 
> will they be hamstrung by the inability to retrieve failure messages that 
> might be informative?
> Maybe everything will be in the logs anyhow?
>
> Mark
>
> -Original Message-
> From: Marlon Pierce [mailto:marpi...@iu.edu]
> Sent: Wednesday, May 14, 2014 12:23 PM
> To: dev@airavata.apache.org
> Subject: Re: Validation failures and storing the failures
>
> I don't think these need to be stored, just sent back to the gateway. 
> If I recall correctly, steps are
>
> 1. Gateway invokes "execute" method of API server.
>
> 2. API Server runs Orchestrator's "validate()".
>
> 3. Orch returns result of validation.
>
> 4. API server returns results to the gateway.
>
> 5. If validation passed, API Server then runs the Orchestrator's "launch()".
>
> So step #4 should enough.
>
> Marlon
>
>
> On 5/14/14 12:17 PM, Lahiru Gunathilake wrote:
>> Hi All,
>>
>> We recently added validation logics to orchestrator, so gateway 
>> developers can add/configure their validators to be invoked. If they 
>> failed they can wrap an error message in their validation logic and return 
>> that.
>> Orchestrator needs to take these error messages and store them to a 
>> persistent storage so that if the validation failed these errors can 
>> be showed to the gateway user to make it correct. Unless we give a 
>> proper error message to the end user there is no way to make it correct.
>>
>> Currently in the orchestrator implementation it does not store the 
>> error but we have a proper result object which keeps the validation 
>> state and a message.
>>
>> Where should I save this information in our data model ? This is an 
>> information specific to each invocation and there could be multiple 
>> error messages came from multiple validators because we do not return 
>> immediately after the first validation failure but rather we invoke 
>> all the validators and collect all the errors (currently we just have 
>> access to them and simply logging those errors in server side). If 
>> there is no placeholder for these validation errors can we add 
>> something to the data model to store them and retrieve them from the client 
>> code ?
>>
>> I have added FAQ type of a document for Orchestrator[1], please 
>> provide your feedback on this.
>>
>> [1]
>> https://docs.google.com/document/d/1FOc0X6HCMZ9E-fTnZQ8aGo7tK20C8lvKl
>> y
>> cNouswICI/edit
>>
>> Regards
>> Lahiru
>>

RE: JGlobus 2.1.0 testing for Bouncy Castle updates

[Miller, Mark]

I was a little confused by this. 
You meant jglobus 2.1, not 1.49 right? its jglobus 2.1, with bc 1.49?

CIPRES has not been able to use jglobus with gridftp because of a bug that 
Terri has described on the list.
I am not sure if this bug is relevant for Airavata, but warn you all of its 
existence here.

It seems from reading the list there is work to do on the jglobus release 
process before further releases are made.

Mark

-Original Message-
From: Suresh Marru [mailto:sma...@apache.org] 
Sent: Wednesday, April 23, 2014 8:46 AM
To: Airavata Dev; Shahbaz Memon
Subject: JGlobus 2.1.0 testing for Bouncy Castle updates

Hi All,

I started a thread  [1] in jglobus mailing list to ping on plans for a 2.1.0 
release which will bump the bouncy castle version to 1.49.

Since this will ease the integration of BES provider with Airavata, I encourage 
you all chime in the discussion and help test and speed the release of jglobus 
1.49. The mailing list to discuss this is jglobus-u...@googlegroups.com

And the code repository is  https://github.com/jglobus/JGlobus

Suresh
[1] - https://groups.google.com/forum/#!topic/jglobus-user/E_dotfVzUX8

RE: Draft program for February hackathon

[Miller, Mark]

In looking at the agenda, and the proposed travel time, I have the following 
questions:

1. What is the timeline for getting the prerequisites done; since if they 
aren't done, we could be hamstrung in accomplishing the goals of the meeting.
(be sure the meeting has the potential for impact)

2. To me items 1 and 2 seem ideally suited to a pre-develop-athon video-cons; 
it would allow us to assimilate this information on a slower time scale, and 
still be fresh to attack session 3 and 4 first thing. Items 3 and 4 seem to be 
particularly appropriate for a joint face to face meeting.
(take care of as much business ahead of time as possible, so we can focus the 
fresh horses on tasks that only be done in this mode)

thoughts?

Mark

-Original Message-
From: Marlon Pierce [mailto:marpi...@iu.edu] 
Sent: Friday, January 31, 2014 6:06 AM
To: dev@airavata.apache.org
Subject: Draft program for February hackathon

Hi all--

Here is a draft agenda for a group hackathon:
https://docs.google.com/document/d/15RWz15uAULigStRnkQX2FQk6Y_av0OU_m1B2wxSUVvg/edit

We have no problem sending an IU contingent to SDSC, and Gary can travel as 
well.  2 days during the last week in February work for many of us.

I want to use everyone's time efficiently.  Hackathons need to be just 
that--hacking, looking at code, getting things to work, etc--so this is what we 
have in mind.

Marlon

RE: Users and UserGroups in Science Gateways

[Miller, Mark]

Thanks for explaining Suresh!

-Original Message-
From: Suresh Marru [mailto:sma...@apache.org] 
Sent: Friday, January 31, 2014 11:55 AM
To: Airavata Dev
Subject: Re: Users and UserGroups in Science Gateways

Let me see if I can clarify:

Airavata expects clients (or gateways) to manage users and groups. And Airavata 
relies on gateway portal request (which it will blindly trust-based on well 
secure communications) to tell which user and group is making a request. The 
discussion I see now happening is for Airavata to manage the input and output 
data. So it will need its own internal structure on how to manage these. 

If a gateway choose to, it can leverage this structure, but this will not be 
required. 

As for the data model:

I suggest against making any implicit derivations of the data models these 
leads to ambiguity once the motivation is forgotten. I rather suggest making 
explicit changes to the data model if the need is justified.  

Airavata data model as I understand is as follows:

Gateways-> Users-> Projects->Experiments

I think what we are talking is to change it induce a groups in between which is 
a fine addition, so then it should be:

Gateways-> Groups-> Users-> Projects-> Experiments.

Simpler use cases may choose to always use a single default group and a user 
may choose to have single project. In that case, it shows like:
Gateways->Users->Experiments, but in the backend its always should have a 
self-explanatory hierarchy. 

Suresh


On Jan 31, 2014, at 2:49 PM, Miller, Mark  wrote:

> I think we agreed that the Gateways would be responsible for user management, 
> but maybe that is at a different level?
>  
> From: Sachith Withana [mailto:swsach...@gmail.com] 
> Sent: Friday, January 31, 2014 11:46 AM
> To: dev@airavata.apache.org
> Subject: Re: Users and UserGroups in Science Gateways
>  
> Thanks Raman,
> I looked into the tables. It looks to be a viable option.
> But is it okay to manage the users from the Airavata server itself?
>  
> 
> On Fri, Jan 31, 2014 at 1:03 PM, Raminder Singh  
> wrote:
> Looking at Airavata Data model, there is project which is equivalent to User 
> group. A project can have multiple user for a gateway and a gateway can have 
> multiple projects.
>  
> Thanks
> Raminder
>  
> On Jan 31, 2014, at 12:44 PM, Saminda Wijeratne  wrote:
> 
> 
>  
>  
> 
> On Fri, Jan 31, 2014 at 9:28 AM, Miller, Mark  wrote:
> As an end user, I think even with just a single tool, the argument for a 
> group is there.
> As Saminda suggested, one may wish to have a join area to work in with a 
> group, and individual work you aren't ready to share, even with just a single 
> tool.
>  
> Because a Gateway is wide open, it is easy to imagine a case where there are 
> several groups, each with (potentially overlapping) users, who all are 
> working with Amber for different projects..
> Agreed. I totally forgot about shared projects scenario. Thanks for pointing 
> it out Mark. 
>  
> Mark
>  
> From: Saminda Wijeratne [mailto:samin...@gmail.com] 
> Sent: Friday, January 31, 2014 9:11 AM
> To: dev
> Subject: Re: Users and UserGroups in Science Gateways
>  
> I'm not an expert on the Amber usecase, but I suppose it is possible to have 
> multiple user groups for a single usecase. It could be to have a shared space 
> of data hidden from each user group. It could be to author 
> access/privilledges to the portal or resources depending on the user group.
> 
> I would assume the mapping of user to usergroups would be many to one 
> mapping. Unlike user roles I don't think it is sensible to have users being 
> mapped to multiple groups (there is a whole area of security to consider 
> here).
>  
> (I do have to note that usergroup IMO is not a must concept for a science 
> gateway)
>  
> 
> On Fri, Jan 31, 2014 at 8:55 AM, Sachith Withana  wrote:
> Hi all,
>  
> Can someone explain to me the mapping of users to groups in a Science 
> Gateway? ( This is regarding the Amber usecase).
>  
> Can there be multiple user groups in a gateway portal when it comes to just 
> one usecase?
> ex: If we have only one usecase for the portal ( such as Amber)
>  
> --
> Thanks,
> Sachith Withana
> 
>  
>  
>  
> 
> 
>  
> --
> Thanks,
> Sachith Withana
> 

RE: Users and UserGroups in Science Gateways

[Miller, Mark]

I think we agreed that the Gateways would be responsible for user management, 
but maybe that is at a different level?

From: Sachith Withana [mailto:swsach...@gmail.com]
Sent: Friday, January 31, 2014 11:46 AM
To: dev@airavata.apache.org
Subject: Re: Users and UserGroups in Science Gateways

Thanks Raman,
I looked into the tables. It looks to be a viable option.
But is it okay to manage the users from the Airavata server itself?

On Fri, Jan 31, 2014 at 1:03 PM, Raminder Singh 
mailto:raminderjsi...@gmail.com>> wrote:
Looking at Airavata Data model, there is project which is equivalent to User 
group. A project can have multiple user for a gateway and a gateway can have 
multiple projects.

Thanks
Raminder

On Jan 31, 2014, at 12:44 PM, Saminda Wijeratne 
mailto:samin...@gmail.com>> wrote:




On Fri, Jan 31, 2014 at 9:28 AM, Miller, Mark 
mailto:mmil...@sdsc.edu>> wrote:
As an end user, I think even with just a single tool, the argument for a group 
is there.
As Saminda suggested, one may wish to have a join area to work in with a group, 
and individual work you aren't ready to share, even with just a single tool.

Because a Gateway is wide open, it is easy to imagine a case where there are 
several groups, each with (potentially overlapping) users, who all are working 
with Amber for different projects..
Agreed. I totally forgot about shared projects scenario. Thanks for pointing it 
out Mark.

Mark

From: Saminda Wijeratne [mailto:samin...@gmail.com<mailto:samin...@gmail.com>]
Sent: Friday, January 31, 2014 9:11 AM
To: dev
Subject: Re: Users and UserGroups in Science Gateways

I'm not an expert on the Amber usecase, but I suppose it is possible to have 
multiple user groups for a single usecase. It could be to have a shared space 
of data hidden from each user group. It could be to author access/privilledges 
to the portal or resources depending on the user group.
I would assume the mapping of user to usergroups would be many to one mapping. 
Unlike user roles I don't think it is sensible to have users being mapped to 
multiple groups (there is a whole area of security to consider here).

(I do have to note that usergroup IMO is not a must concept for a science 
gateway)

On Fri, Jan 31, 2014 at 8:55 AM, Sachith Withana 
mailto:swsach...@gmail.com>> wrote:
Hi all,

Can someone explain to me the mapping of users to groups in a Science Gateway? 
( This is regarding the Amber usecase).

Can there be multiple user groups in a gateway portal when it comes to just one 
usecase?
ex: If we have only one usecase for the portal ( such as Amber)

--
Thanks,
Sachith Withana






--
Thanks,
Sachith Withana

RE: Users and UserGroups in Science Gateways

[Miller, Mark]

In the CIPRES usecase, how would you allow access to shared spaces in terms of 
the user experience?
having one shared Space for the user and populate with all the experiments 
shared with the user or
having multiple shared spaces categorized by the usergroup?

The current CIPRES use case is focused on the single user, and so I would 
imagine the former.
This isn't an issue we have addressed yet since we don't support groups yet. 
But I imagine it would be
the users space is populated with all items (s)he is allowed to see.



From: Sachith Withana [mailto:swsach...@gmail.com]
Sent: Friday, January 31, 2014 9:19 AM
To: dev@airavata.apache.org
Subject: Re: Users and UserGroups in Science Gateways

Thanks a lot Saminda,

So a user will have access to multiple shared spaces ( for each usergroup) and 
to his/her own private space.

In the CIPRES usecase, how would you allow access to shared spaces in terms of 
the user experience?
having one shared Space for the user and populate with all the experiments 
shared with the user or
having multiple shared spaces categorized by the usergroup?


On Fri, Jan 31, 2014 at 12:11 PM, Saminda Wijeratne 
mailto:samin...@gmail.com>> wrote:
I'm not an expert on the Amber usecase, but I suppose it is possible to have 
multiple user groups for a single usecase. It could be to have a shared space 
of data hidden from each user group. It could be to author access/privilledges 
to the portal or resources depending on the user group.
I would assume the mapping of user to usergroups would be many to one mapping. 
Unlike user roles I don't think it is sensible to have users being mapped to 
multiple groups (there is a whole area of security to consider here).

(I do have to note that usergroup IMO is not a must concept for a science 
gateway)

On Fri, Jan 31, 2014 at 8:55 AM, Sachith Withana 
mailto:swsach...@gmail.com>> wrote:
Hi all,

Can someone explain to me the mapping of users to groups in a Science Gateway? 
( This is regarding the Amber usecase).

Can there be multiple user groups in a gateway portal when it comes to just one 
usecase?
ex: If we have only one usecase for the portal ( such as Amber)

--
Thanks,
Sachith Withana




--
Thanks,
Sachith Withana

RE: Users and UserGroups in Science Gateways

[Miller, Mark]

As an end user, I think even with just a single tool, the argument for a group 
is there.
As Saminda suggested, one may wish to have a join area to work in with a group, 
and individual work you aren't ready to share, even with just a single tool.

Because a Gateway is wide open, it is easy to imagine a case where there are 
several groups, each with (potentially overlapping) users, who all are working 
with Amber for different projects..

Mark

From: Saminda Wijeratne [mailto:samin...@gmail.com]
Sent: Friday, January 31, 2014 9:11 AM
To: dev
Subject: Re: Users and UserGroups in Science Gateways

I'm not an expert on the Amber usecase, but I suppose it is possible to have 
multiple user groups for a single usecase. It could be to have a shared space 
of data hidden from each user group. It could be to author access/privilledges 
to the portal or resources depending on the user group.
I would assume the mapping of user to usergroups would be many to one mapping. 
Unlike user roles I don't think it is sensible to have users being mapped to 
multiple groups (there is a whole area of security to consider here).

(I do have to note that usergroup IMO is not a must concept for a science 
gateway)

On Fri, Jan 31, 2014 at 8:55 AM, Sachith Withana 
mailto:swsach...@gmail.com>> wrote:
Hi all,

Can someone explain to me the mapping of users to groups in a Science Gateway? 
( This is regarding the Amber usecase).

Can there be multiple user groups in a gateway portal when it comes to just one 
usecase?
ex: If we have only one usecase for the portal ( such as Amber)

--
Thanks,
Sachith Withana

RE: [VOTE] Convert from SVN to GIT

[Miller, Mark]

+1
mark

From: Amila Jayasekara [mailto:thejaka.am...@gmail.com]
Sent: Tuesday, January 21, 2014 7:56 AM
To: dev
Subject: Re: [VOTE] Convert from SVN to GIT

+1.

Thanks
Amila

On Tue, Jan 21, 2014 at 10:08 AM, Marlon Pierce 
mailto:marpi...@iu.edu>> wrote:
This is the [VOTE] thread for changing Airavata's version control system
from SVN to Git.  See [1] for background.  Please vote as follows:

+1 if you are in favor of changing from SVN to GIT

+/- 0 if you want to express no opinion

-1 if you oppose the change

The voting thread will be open for 72 hours.  Please use the
accompanying [DISCUSS] thread for any discussion.


Thanks--

Marlon

[1]
http://mail-archives.apache.org/mod_mbox/airavata-dev/201401.mbox/%3C52D83811.8000209%40iu.edu%3E