Hi Saminda, There are two major points we need to address to answer your question.
1. who are we trying to cater and what they want 2. once we know what we want from #1 above how do we achieve it #1: We can either make it easy for the scientists who use the gateway or make it easy for the developer, i.e. you. I think one of the major selling points for science gateways like Airavata is that it will make the life of a non-computer scientist easy by abstracting away the complexities of gateways. A common example we used to mention when we were in IU is that Sun Kim (a bio scientist in IU) should better spend his time trying to improve his algorithms than trying to tackle with WS-GRAM. Also, he should not worry too much when he decide to run an application in BigRed (if its still there :) ) vs NCSA vs Prof. Fox's windows HPC cluster. It should ideally be just a matter of flipping a switch in the API (but yes there will be some pre-preparation, like installing apps, before doing this). I think we should give priority to make it easy for the clients. So, if we agree on this with the conclusion that the scientists, i.e. Airavata clients are hidden from the underlying job submission mechanisms then we can tackle #2. #2: When I was working with Sigiri (in my previous life) I had the same problem. I had two main methods to accomplish that. 1. I looked into solutions like JSDL ( http://en.wikipedia.org/wiki/Job_Submission_Description_Language) which had already come up with an abstraction. But this was kind of biased towards Grids. 2. I looked into some of the existing gateways to define a generic API At the end, I ended up defining a generic API from what I learnt from both methods but making sure the client API is simple. The API 1. simple enough to submit simple jobs that are covering about 80% of the use cases (including parametric sweeps) 2. extensible enough to cover the corner cases which amounted to about 20% of the use cases Within the implementations of each gateway, I had Adapters for each gateway and were picked based on what clients need. I can go into details if needed (and I need to go back to code) but this is something worth considering. BTW, I understand the fact that the Airavata thrift API may be "hidden" underneath XBaya but I'm sure there will be usecases where some scientists (and their poor grad students) wanting to launch workflows programmatically using thrift API. Thanks, Eran Chinthaka Withana On Thu, Jan 30, 2014 at 8:54 PM, Saminda Wijeratne <[email protected]>wrote: > I'm thinking of the reasons why we are having trouble getting a proper API > to work smoothly is that, > > > 1. We are trying too hard to generalize when its probably not possible > to do > 2. When we keep trying to see the big picture we miss the little things > in the gateways that makes the usage of the API easy or hard for the > developer. > > eg: Not all gateways look at "Experiment Launch" use case as passing an > application name and its input to launch the experiment. (I think we've > came across many variations for this use case) > > - the inputs could be in different form (could be actual files) or just > optional > - application name could be just a reference to a parameter of a global > application > - part of the launch process may have runtime dependencies which gateway > requires to participate > - the gateway might want more control over the launch configurations at > runtime. > - ... > > I think devs see it as a mismatch of the API for their gateway which > discourages the effort to use the API. > > IMO part of our Thrift API effort should be to document these with the > solutions similar to Java Patterns. > > In short, instead of trying to come up with a general API to make everyone > happy we provide solution patterns for a gateway use case which will have > multiple variations. (solution pattern would be use of the API + supporting > code which handles the variation) > > wdyt? > > > Regards, > > Saminda >
