Except for HadoopProvider I've completed job data persistance for all the other 5 providers. (We should revisit it after the release to verify the implementation)
On Fri, Jun 14, 2013 at 11:13 AM, Suresh Marru <[email protected]> wrote: > On Jun 14, 2013, at 10:49 AM, Lahiru Gunathilake <[email protected]> > wrote: > > > Hi Marlon, > > > > I have no update about the release, since I've been working with Gary in > > Ultrascan project. I found some issues to fix for ultrascan, now working > on > > them, so we have to wait until Gary gives the green light to do the > release. > > Lahiru, thanks for all your work on the release blocks and for being the > RM for this release. > > No biggie for this case but want to caution: > > Just a friendly reminder to all of us - 'if it did not happen in the > mailing list, it did not happen'. So please do not assume context with > these discussions and explicitly state the issues. Also, ultrascan cannot > block airavata release. If Gary or others report issues to you offline from > a RC-* testing, please alert the list or raise a JIRA (thanks for doing it > for this issue). > > Suresh > > > > > WDYT ? > > > > Regards > > Lahiru > > > > > > On Fri, Jun 14, 2013 at 10:44 AM, Marlon Pierce <[email protected]> wrote: > > > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA1 > >> > >> What's the status of the 0.8 release? > >> > >> > >> Marlon > >> > >> > >> On 5/31/13 11:53 AM, Saminda Wijeratne wrote: > >>> On Fri, May 31, 2013 at 11:31 AM, Amila Jayasekara > >>> <[email protected]>wrote: > >>> > >>>> On Fri, May 31, 2013 at 11:26 AM, Saminda Wijeratne > >>>> <[email protected] > >>>>> wrote: > >>>> > >>>>> Latest update on AiravataAPI updates for 0.8 version > >>>>> > >>>>> 1. *Error message from the immediate exception used as the > >>>>> error > >>>> message > >>>>> for the AiravataAPInvocationException*. Earlier it was just > >>>>> "Error invoking API". *This change was made so that the users > >>>>> of the API can show the error from the exception without having > >>>>> to dig in to the > >>>> inner > >>>>> exception to get a better error message.* 2. *Saving and > >>>>> retrieving experiment execution error details*. Earlier we were > >>>>> not persisting any execution error details of the workflow. > >>>>> Error details are only available only if the developers are > >>>>> monitoring experiment at the time of error occurs. *This is not > >>>>> feasible if users would only want to go through the error > >>>>> details later without doing monitoring > >>>> from > >>>>> the beginning. Therefore we are now persisting the error > >>>>> details (not > >>>> as > >>>>> part of provenance data) of executing workflows (i.e. > >>>>> experiments) in > >>>> to > >>>>> the registry and also retrieving from registry through Airavata > >>>>> API (which calls the Registry API). These functions are > >>>>> available in the "ExecutionManager" of the AiravataAPI and > >>>>> "ProvenanceRegistry" in the RegistryAPI. * 3. Persisting & > >>>>> retrieving GFac job data. There are many benefits of saving the > >>>>> GRAM data, EC2 data etc. such as for debugging, analysis > >>>> etc. > >>>>> Currently these data are available only through the message > >>>>> notifications. *It is not feasible for the same reasons > >>>>> mentioned for API change for saving error data. Hence the > >>>>> introduction of functions in the API to save & retrieve GFac > >>>>> job data.** These functions are available as > >>>>> get/setApplicationJob***(...) functions in the > >>>>> "ProvenanceManager" of the AiravataAPI and "ProvenanceRegistry" > >>>>> in the RegistryAPI.* > >>>>> > >>>> > >>>> May I ask why we need setApplicationJob****() methods in the API > >>>> ? As far as I understood job information is populated by GFac and > >>>> other internal components. So why do we need API methods to set > >>>> those data ? (OR am I missing something ?) > >>>> > >>> GFac and Internal components also use the AiravataAPI to populate > >>> the job data in the registry. > >>> > >>> > >>>> Thanks Amila > >>>> > >>>> > >>>>> > >>>>> > >>>>> Regards, > >>>>> > >>>>> Saminda > >>>>> > >>>>> > >>>>> On Tue, May 21, 2013 at 4:59 PM, Raminder Singh < > >>>> [email protected] > >>>>>> wrote: > >>>>> > >>>>>> Thanks Chathuri, I will test and let you know. > >>>>>> > >>>>>> Raman On May 21, 2013, at 4:55 PM, Chathuri Wimalasena > >>>>>> wrote: > >>>>>> > >>>>>>> Hi Raman, > >>>>>>> > >>>>>>> The method Saminda mentioned will work for your case. I > >>>>>>> tested with > >>>>> your > >>>>>>> test class. I fixed deserialization issues from REST > >>>>>>> service side. > >>>> You > >>>>>> will > >>>>>>> probably have to update both server and client. > >>>>>>> > >>>>>>> Regards, Chathuri > >>>>>>> > >>>>>>> > >>>>>>> On Tue, May 21, 2013 at 3:04 PM, Saminda Wijeratne < > >>>> [email protected] > >>>>>>> wrote: > >>>>>>> > >>>>>>>> So far what you are saying is true. If the client knows > >>>>>>>> which node > >>>>>> failed > >>>>>>>> he/she can get the list of errors for that node using > >>>>>>>> the getExecutionError(...) function. Use the "type" as > >>>>>>>> ALL and leave the gfacJobId as null. I couldn;t think of > >>>>>>>> doing this any other better > >>>> way > >>>>>> than > >>>>>>>> introducing a whole lot of functions which could be more > >>>>>> annoying/confusing > >>>>>>>> to figure-out. I'm welcome for suggestions for > >>>>>>>> improvements. > >>>>>>>> > >>>>>>>> Saminda > >>>>>>>> > >>>>>>>> > >>>>>>>> On Tue, May 21, 2013 at 2:36 PM, Raminder Singh < > >>>>>> [email protected] > >>>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Thanks Chathuri, the WorkflowExecutionError works. > >>>>>>>>> Integrating to > >>>>>> Error > >>>>>>>>> API brought few questions. We have ErrorTypes as : > >>>>>>>> WorkflowExecutionError, > >>>>>>>>> ExperimentExecutionError, NodeExecutionError, > >>>> GFacJobExecutionError, > >>>>>>>>> ExecutionError . How do we want clients to know which > >>>>>>>>> error type to > >>>>>>>> query? > >>>>>>>>> I ran a job and it failed, now i want to get error. > >>>>>>>>> Error occurred > >>>> at > >>>>>>>> GFac > >>>>>>>>> provider level and client does not have any > >>>>>>>>> information. Do we > >>>> expect > >>>>>>>>> client to query all different types? > >>>>>>>>> > >>>>>>>>> I thing we can have is a method to get all error and > >>>>>>>>> return the > >>>> Error > >>>>>>>> type > >>>>>>>>> part of the object to help the client to classify the > >>>>>>>>> error > >>>> location. > >>>>>>>>> Other suggestions? > >>>>>>>>> > >>>>>>>>> Raminder > >>>>>>>>> > >>>>>>>>> On May 21, 2013, at 10:57 AM, Chathuri Wimalasena > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hi Raman, > >>>>>>>>>> > >>>>>>>>>> WorkflowExecutionError method not returning any data > >>>>>>>>>> issue should > >>>> be > >>>>>>>>> fixed > >>>>>>>>>> now. Can you get an update and check. > >>>>>>>>>> > >>>>>>>>>> Regards, Chathuri > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Tue, May 21, 2013 at 10:02 AM, Raminder Singh > >>>>>>>>>> <[email protected]>wrote: > >>>>>>>>>> > >>>>>>>>>>> Thank, these are very useful for API users. Only > >>>>>>>>>>> thing i found > >>>>>>>> confusing > >>>>>>>>>>> for the user is API managers. ProvenanceManager is > >>>>>>>>>>> the one used > >>>> to > >>>>>>>> get > >>>>>>>>>>> experiment status. Users will get FINISHED or > >>>>>>>>>>> FAILED status from > >>>>> the > >>>>>>>>> data > >>>>>>>>>>> object. On failure, i was trying to find a method > >>>>>>>>>>> to get the > >>>> error > >>>>>>>> data > >>>>>>>>> but > >>>>>>>>>>> i was not able to find as those methods exist in > >>>> ExecutionManager. > >>>>>>>>>>> According to me, experiment status and in case of > >>>>>>>>>>> failures get > >>>>> error > >>>>>>>>> need > >>>>>>>>>>> to be part of one manager. May be a > >>>>>>>>>>> ExperimentManager? Getting > >>>>>> output > >>>>>>>>> data > >>>>>>>>>>> can be left as part of provenance manages as only > >>>>>>>>>>> few client may > >>>>> want > >>>>>>>> to > >>>>>>>>>>> get the data in few cases. Getting error detail in > >>>>>>>>>>> case of error > >>>> is > >>>>>>>>> obvious > >>>>>>>>>>> step. > >>>>>>>>>>> > >>>>>>>>>>> Other thing can you please explain the difference > >>>>>>>>>>> between WorkflowExecutionError, > >>>>>>>>>>> ExperimentExecutionError, > >>>>> NodeExecutionError, > >>>>>>>>>>> GFacExecutionError on a Wiki page for the users. > >>>>>>>>>>> Also add a > >>>> method > >>>>> to > >>>>>>>>> get > >>>>>>>>>>> JobID as that is required to get > >>>>>>>>>>> GFacExecutionError. > >>>>>>>>>>> > >>>>>>>>>>> I tested the WorkflowExecutionError method and its > >>>>>>>>>>> not returning > >>>>> any > >>>>>>>>> data. > >>>>>>>>>>> Can you please test that as well? > >>>>>>>>>>> > >>>>>>>>>>> ExperimentData data = > >>>>>>>>>>> > >>>> airavataAPI.getProvenanceManager().getExperimentData(experimentID); > >>>>>>>>>>> > >>>> > >> List<WorkflowExecutionError> experimentExecutionErrors = > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > >>>> > >> > airavataAPI.getExecutionManager().getWorkflowExecutionErrors(experimentID, > >>>>>>>>>>> > >>>> > >> experimentID); > >>>>>>>>>>> > >>>>>>>>>>> Thanks Raminder > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On May 20, 2013, at 5:05 PM, Saminda Wijeratne > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Following updates will be present Airavata API > >>>>>>>>>>>> from 0.8 release. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> 1. *Error message from the immediate exception > >>>>>>>>>>>> used as the error > >>>>>>>>>>> message > >>>>>>>>>>>> for the AiravataAPInvocationException*. Earlier > >>>>>>>>>>>> it was just > >>>> "Error > >>>>>>>>>>>> invoking API". *This change was made so that the > >>>>>>>>>>>> users of the > >>>> API > >>>>>>>> can > >>>>>>>>>>>> show the error from the exception without having > >>>>>>>>>>>> to dig in to > >>>> the > >>>>>>>>> inner > >>>>>>>>>>>> exception to get a better error message.* 2. > >>>>>>>>>>>> *Saving and retrieving experiment execution error > >>>>>>>>>>>> details*. > >>>>>>>> Earlier > >>>>>>>>>>>> we were not persisting any execution error > >>>>>>>>>>>> details of the > >>>>> workflow. > >>>>>>>>>>> Error > >>>>>>>>>>>> details are only available only if the developers > >>>>>>>>>>>> are monitoring > >>>>>>>>>>> experiment > >>>>>>>>>>>> at the time of error occurs. *This is not > >>>>>>>>>>>> feasible if users > >>>> would > >>>>>>>> only > >>>>>>>>>>>> want to go through the error details later > >>>>>>>>>>>> without doing > >>>>> monitoring > >>>>>>>>>>> from > >>>>>>>>>>>> the beginning. Therefore we are now persisting > >>>>>>>>>>>> the error details > >>>>>>>> (not > >>>>>>>>>>> as > >>>>>>>>>>>> part of provenance data) of executing workflows > >>>>>>>>>>>> (i.e. > >>>>> experiments)* > >>>>>>>> in > >>>>>>>>>>>> to the registry and also retrieving from registry > >>>>>>>>>>>> through > >>>> Airavata > >>>>>>>> API > >>>>>>>>>>>> (which calls the Registry API). > >>>>>>>>>>>> > >>>>>>>>>>>> For now we have the functions for saving and > >>>>>>>>>>>> retrieving errors > >>>> in > >>>>>> the > >>>>>>>>>>>> ExecutionManager[1]. We need to decide if this is > >>>>>>>>>>>> the correct > >>>>> place > >>>>>>>> to > >>>>>>>>>>> put > >>>>>>>>>>>> these functions. Your thoughts in this matter are > >>>>>>>>>>>> most > >>>> welcome... > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, Saminda > >>>>>>>>>>>> > >>>>>>>>>>>> 1. > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > >>>> > >> > https://svn.apache.org/repos/asf/airavata/trunk/modules/airavata-client/src/main/java/org/apache/airavata/client/api/ExecutionManager.java > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> -----BEGIN PGP SIGNATURE----- > >> Version: GnuPG/MacGPG2 v2.0.18 (Darwin) > >> Comment: GPGTools - http://gpgtools.org > >> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > >> > >> iQEcBAEBAgAGBQJRuyxqAAoJEOEgD2XReDo54nwIAKjDjVqPk6YldnJD+T91k2HU > >> Yj/VcvH/XAxvoZOu9zGpcUFNuLPbSiJEJes9Tx3Wk63k8v+l4rxTTRB5rWihbyYn > >> xPEKtSnD8sCAiuFXlRHWLygzE80OVhC9Q43xkrnjjubKaWKp3ZgU1uF1ZL6LiTZb > >> R83+Yl2WCdAAckX1W+567APk202nw93GLKGjmRcuj2mBPKgWXzzD7AX3cQX6pt9e > >> g93PKGWJWdk/kb6YlhjkEfKH2RxJS/AgZvUYSAmf1ELL+siGqxEEOg7bwWwpfLNd > >> NsjUr/rN88EcGfUKxYb4fSh1zGceqp02rWG2TwufZEIVaF8WjztY6CsE2TgfS0M= > >> =vpMl > >> -----END PGP SIGNATURE----- > >> > > > > > > > > -- > > System Analyst Programmer > > PTI Lab > > Indiana University > >
