I agree with your argument on identifying the focus groups and their needs and figuring out how to solve their problems. Figuring out a simple API is indeed important. My concern was more towards the next step. How to present our solution to the users. Having a well-defined API covering the required functionalities not necessarily can be made simple enough to be understood or doesn't explain most of the time how to arrive at a point where they can be used. For instance designing a function to retrieve experiment data requires us to understand atleast the level of granularity of the data expected, filtrations to be considered and the frequency of invocations.
On the few gateways I've worked so far (ParamChem, Ultrascan and CIPRES/NSG) I didn't see the direct usage of the current API. While in each case API showed alot of strength in certain areas where it made developer life really easy, most of the time they had to go through a few hops to actually to reach the use of API (and most of the time this is with our help). These hops are not always obvious and time it takes to figuring them out could be the difference between giving up or making a commitment to learn Airavata API nooks and corners. My point is that we can reduce this time by documenting what we already know on different variations of the same use case. My recent experience on CIPRES + Airavata integration is a good example why I'm proposing this. CIPRES has a different pluggable framework of handling jobs and their data compared to Airavata. It took me a while to understand how Airavata can be plugged in to CIPRES even with my knowledge of Airavata and (limited though) CIPRES. Eventhough it wouldn't be an ideal way of integrating Airavata to CIPRES (one too many levels of abstractions) the resultant implementation is infact simple enough to understand to a gateway dev. But the fact is I had to spend alot of time getting to that solution. It was the same case with the ParamChem integration (although it took less time). On Thu, Jan 30, 2014 at 10:18 PM, Eran Chinthaka Withana < [email protected]> wrote: > Hi Saminda, > > There are two major points we need to address to answer your question. > > 1. who are we trying to cater and what they want > 2. once we know what we want from #1 above how do we achieve it > > #1: We can either make it easy for the scientists who use the gateway or > make it easy for the developer, i.e. you. I think one of the major selling > points for science gateways like Airavata is that it will make the life of > a non-computer scientist easy by abstracting away the complexities of > gateways. A common example we used to mention when we were in IU is that > Sun Kim (a bio scientist in IU) should better spend his time trying to > improve his algorithms than trying to tackle with WS-GRAM. Also, he should > not worry too much when he decide to run an application in BigRed (if its > still there :) ) vs NCSA vs Prof. Fox's windows HPC cluster. It should > ideally be just a matter of flipping a switch in the API (but yes there > will be some pre-preparation, like installing apps, before doing this). I > think we should give priority to make it easy for the clients. So, if we > agree on this with the conclusion that the scientists, i.e. Airavata > clients are hidden from the underlying job submission mechanisms then we > can tackle #2. > > #2: When I was working with Sigiri (in my previous life) I had the same > problem. I had two main methods to accomplish that. > 1. I looked into solutions like JSDL ( > http://en.wikipedia.org/wiki/Job_Submission_Description_Language) which > had > already come up with an abstraction. But this was kind of biased towards > Grids. > 2. I looked into some of the existing gateways to define a generic API > > At the end, I ended up defining a generic API from what I learnt from both > methods but making sure the client API is simple. The API > 1. simple enough to submit simple jobs that are covering about 80% of the > use cases (including parametric sweeps) > 2. extensible enough to cover the corner cases which amounted to about 20% > of the use cases > > Within the implementations of each gateway, I had Adapters for each gateway > and were picked based on what clients need. I can go into details if needed > (and I need to go back to code) but this is something worth considering. > > BTW, I understand the fact that the Airavata thrift API may be "hidden" > underneath XBaya but I'm sure there will be usecases where some scientists > (and their poor grad students) wanting to launch workflows programmatically > using thrift API. > > Thanks, > Eran Chinthaka Withana > > > On Thu, Jan 30, 2014 at 8:54 PM, Saminda Wijeratne <[email protected] > >wrote: > > > I'm thinking of the reasons why we are having trouble getting a proper > API > > to work smoothly is that, > > > > > > 1. We are trying too hard to generalize when its probably not possible > > to do > > 2. When we keep trying to see the big picture we miss the little > things > > in the gateways that makes the usage of the API easy or hard for the > > developer. > > > > eg: Not all gateways look at "Experiment Launch" use case as passing an > > application name and its input to launch the experiment. (I think we've > > came across many variations for this use case) > > > > - the inputs could be in different form (could be actual files) or > just > > optional > > - application name could be just a reference to a parameter of a > global > > application > > - part of the launch process may have runtime dependencies which > gateway > > requires to participate > > - the gateway might want more control over the launch configurations > at > > runtime. > > - ... > > > > I think devs see it as a mismatch of the API for their gateway which > > discourages the effort to use the API. > > > > IMO part of our Thrift API effort should be to document these with the > > solutions similar to Java Patterns. > > > > In short, instead of trying to come up with a general API to make > everyone > > happy we provide solution patterns for a gateway use case which will have > > multiple variations. (solution pattern would be use of the API + > supporting > > code which handles the variation) > > > > wdyt? > > > > > > Regards, > > > > Saminda > > >
