Thanks Suresh, just let me know if I get too annoying... -----Original Message----- From: Suresh Marru [mailto:[email protected]] Sent: Friday, January 31, 2014 11:59 AM To: [email protected] Subject: Re: Documenting Airavata API usage patterns
Great points Mark and glad to see you every one's participation in these vital usability discussions. Suresh On Jan 31, 2014, at 10:54 AM, Miller, Mark <[email protected]> wrote: > Although I am non-technical, I would just like to say that (IMO) the > issues Saminda is raising here and our ability to finesse them as a group > will be the single most important determinants of SciGap success or failure > as a project and a software package. > > -----Original Message----- > From: Saminda Wijeratne [mailto:[email protected]] > Sent: Thursday, January 30, 2014 11:35 PM > To: architecture > Subject: Re: Documenting Airavata API usage patterns > > I agree with your argument on identifying the focus groups and their needs > and figuring out how to solve their problems. Figuring out a simple API is > indeed important. My concern was more towards the next step. How to present > our solution to the users. Having a well-defined API covering the required > functionalities not necessarily can be made simple enough to be understood or > doesn't explain most of the time how to arrive at a point where they can be > used. For instance designing a function to retrieve experiment data requires > us to understand atleast the level of granularity of the data expected, > filtrations to be considered and the frequency of invocations. > > On the few gateways I've worked so far (ParamChem, Ultrascan and > CIPRES/NSG) I didn't see the direct usage of the current API. While in each > case API showed alot of strength in certain areas where it made developer > life really easy, most of the time they had to go through a few hops to > actually to reach the use of API (and most of the time this is with our > help). These hops are not always obvious and time it takes to figuring them > out could be the difference between giving up or making a commitment to learn > Airavata API nooks and corners. My point is that we can reduce this time by > documenting what we already know on different variations of the same use case. > > My recent experience on CIPRES + Airavata integration is a good example why > I'm proposing this. CIPRES has a different pluggable framework of handling > jobs and their data compared to Airavata. It took me a while to understand > how Airavata can be plugged in to CIPRES even with my knowledge of Airavata > and (limited though) CIPRES. Eventhough it wouldn't be an ideal way of > integrating Airavata to CIPRES (one too many levels of abstractions) the > resultant implementation is infact simple enough to understand to a gateway > dev. But the fact is I had to spend alot of time getting to that solution. > It was the same case with the ParamChem integration (although it took less > time). > > > On Thu, Jan 30, 2014 at 10:18 PM, Eran Chinthaka Withana < > [email protected]> wrote: > >> Hi Saminda, >> >> There are two major points we need to address to answer your question. >> >> 1. who are we trying to cater and what they want 2. once we know what >> we want from #1 above how do we achieve it >> >> #1: We can either make it easy for the scientists who use the gateway >> or make it easy for the developer, i.e. you. I think one of the major >> selling points for science gateways like Airavata is that it will >> make the life of a non-computer scientist easy by abstracting away >> the complexities of gateways. A common example we used to mention >> when we were in IU is that Sun Kim (a bio scientist in IU) should >> better spend his time trying to improve his algorithms than trying to >> tackle with WS-GRAM. Also, he should not worry too much when he >> decide to run an application in BigRed (if its still there :) ) vs >> NCSA vs Prof. Fox's windows HPC cluster. It should ideally be just a >> matter of flipping a switch in the API (but yes there will be some >> pre-preparation, like installing apps, before doing this). I think we >> should give priority to make it easy for the clients. So, if we agree >> on this with the conclusion that the scientists, i.e. Airavata >> clients are hidden from the underlying job submission mechanisms then we can >> tackle #2. >> >> #2: When I was working with Sigiri (in my previous life) I had the >> same problem. I had two main methods to accomplish that. >> 1. I looked into solutions like JSDL ( >> http://en.wikipedia.org/wiki/Job_Submission_Description_Language) >> which had already come up with an abstraction. But this was kind of >> biased towards Grids. >> 2. I looked into some of the existing gateways to define a generic >> API >> >> At the end, I ended up defining a generic API from what I learnt from >> both methods but making sure the client API is simple. The API 1. >> simple enough to submit simple jobs that are covering about 80% of >> the use cases (including parametric sweeps) 2. extensible enough to >> cover the corner cases which amounted to about 20% of the use cases >> >> Within the implementations of each gateway, I had Adapters for each >> gateway and were picked based on what clients need. I can go into >> details if needed (and I need to go back to code) but this is something >> worth considering. >> >> BTW, I understand the fact that the Airavata thrift API may be "hidden" >> underneath XBaya but I'm sure there will be usecases where some >> scientists (and their poor grad students) wanting to launch workflows >> programmatically using thrift API. >> >> Thanks, >> Eran Chinthaka Withana >> >> >> On Thu, Jan 30, 2014 at 8:54 PM, Saminda Wijeratne >> <[email protected] >>> wrote: >> >>> I'm thinking of the reasons why we are having trouble getting a >>> proper >> API >>> to work smoothly is that, >>> >>> >>> 1. We are trying too hard to generalize when its probably not possible >>> to do >>> 2. When we keep trying to see the big picture we miss the little >> things >>> in the gateways that makes the usage of the API easy or hard for the >>> developer. >>> >>> eg: Not all gateways look at "Experiment Launch" use case as passing >>> an application name and its input to launch the experiment. (I think >>> we've came across many variations for this use case) >>> >>> - the inputs could be in different form (could be actual files) or >> just >>> optional >>> - application name could be just a reference to a parameter of a >> global >>> application >>> - part of the launch process may have runtime dependencies which >> gateway >>> requires to participate >>> - the gateway might want more control over the launch >>> configurations >> at >>> runtime. >>> - ... >>> >>> I think devs see it as a mismatch of the API for their gateway which >>> discourages the effort to use the API. >>> >>> IMO part of our Thrift API effort should be to document these with >>> the solutions similar to Java Patterns. >>> >>> In short, instead of trying to come up with a general API to make >> everyone >>> happy we provide solution patterns for a gateway use case which will >>> have multiple variations. (solution pattern would be use of the API >>> + >> supporting >>> code which handles the variation) >>> >>> wdyt? >>> >>> >>> Regards, >>> >>> Saminda >>> >>
