Hi All, I think this is a good idea, from airavata point of view, it sounds like a good idea to hand over the static data to some other party (static data is the data generated during a job submission and monitoring, which is not goingt o change). Airavata shouldn't get burdened with those provenance type of data. For airavata's runtime we only need these data, until the job get completed.
Based on the usecase of the science gateway, they can arrange their data and make the data retrieval process more reliable. My idea is airavata server or runtime should not get burden with provenance data, if Airavata needs this feature we might have to separate it from Airavata runtime (I mean job submission and monitoring). Regards Lahiru On Fri, Mar 14, 2014 at 12:11 PM, Borries Demeler < [email protected]> wrote: > I am not sure if this relates at all to the getAllExperiments() question, > but I thought it may help to see how we deal with retrieving expt. data > from > the gateway in UltraScan. > > In the case of UltraScan we have the HPC output very much separated from > the > information of the results that are useful to the user. All data are > getting > tar.gzipped into a single archive which is transported back out and > automatically > parsed into a relational database that is part of the UltraScan software, > and not > a function of the gateway. The user will then access the database with > secondary > software and create a type of meta-analysis and visualization of the > results > that ultimately are then useful to the user. Having said this, while data > are > being calculated on the HPC resource, the user can monitor the process of > the > calculation by reviewing a queue viewer that shows all pending and active > jobs > and their actual state. This state include a lot of information keeping the > user apprised of the state. This information is continually being sent via > UDP > from the HPC resource and monitored by a daemon on our backend LIMS server, > again, this is semi-independent from the gateway software. This may be very > specialized for our use case, but it is one example of how the problem can > be > solved. > > -borries > > On Fri, Mar 14, 2014 at 11:37:02AM -0400, Saminda Wijeratne wrote: > > Thanks for identifying the problem Sachith. To be precise potentially > there > > are many subset variations of interested fields in the Experiment Data > > which different gateways will be interested in the same usecase or > > different usecase. We cannot scale by providing different functions for > > each of these scenarios. > > > > > > On Fri, Mar 14, 2014 at 11:25 AM, Sachith Withana <[email protected] > >wrote: > > > > > Hi all, > > > > > > Almost all gateways have a requirement of retrieving the experiment > > > summaries of all the experiments . The fields that are required differ > > > based on the gateway. > > > > > > For Example: > > > CIPRES requires : Experiment name and status > > > Some gateways requires the Experiment name with the inputs only. > > > > > > But right now the getAllExperiments() method returns the list of all > the > > > Experiments with all the experiment related attributes filled ( the > whole > > > Experiment Model). > > > > > > It's costly to get the whole Experiment objects from the API rather > than > > > getting the required few attributes. > > > > > > Any suggestions on how we could achieve this? > > > > > > One suggestion would be to have a getAllExperiments method with the > > > parameters as a list of required fields of the Experiments and return > only > > > those fields. > > > > > > -- > > > Thanks, > > > Sachith Withana > > > > -- System Analyst Programmer PTI Lab Indiana University
